Method and apparatus for inter-layer encoding and method and apparatus for inter-layer decoding video using residual prediction

ABSTRACT

Provided are a method and apparatus for inter-layer encoding and a method and apparatus for inter-layer decoding video using residual prediction. The inter-layer video decoding method includes: obtaining a decoded first layer image from a first layer bitstream; determining, based on residual prediction information obtained from a second layer bitstream, whether to perform residual prediction performed on a second layer block by referring to a first layer block corresponding to the second layer block within the first layer image; in response to determining not to perform the residual prediction, obtaining luminance compensation information from the second layer bitstream and determining whether to perform luminance compensation based on the luminance compensation information; and in response to determining to perform the luminance compensation, performing the luminance compensation on the second layer block.

RELATED APPLICATIONS

This is a national stage application of PCT/KR2014/006327 filed on Jul.14, 2014 which claims the benefit of U.S. Provisional Application61/845,548 filed on Jul. 12, 2013, in the United States Patent andTrademark Office, the disclosures of which are hereby incorporatedherein in their entirety by reference.

BACKGROUND 1. Technical Field

Apparatuses and methods consistent with exemplary embodiments relate tointer-layer video encoding and decoding methods, and more particularly,to a method of performing residual prediction between inter-layerimages.

2. Related Art

As hardware for reproducing and storing high resolution or high qualityvideo content is being developed and supplied, a need for a video codecfor effectively encoding or decoding the high resolution or high qualityvideo content is increasing. According to a video codec, a video isencoded according to a limited encoding method based on a coding unithaving a predetermined size.

Image data of the space domain is transformed into coefficients of thefrequency domain via frequency transformation. According to a videocodec, an image is split into blocks having a predetermined size,discrete cosine transformation (DCT) is performed on each block, andfrequency coefficients are encoded in block units, for rapid calculationof frequency transformation. Compared with image data of the spacedomain, coefficients of the frequency domain are easily compressed. Inparticular, since an image pixel value of the space domain is expressedaccording to a prediction error via inter prediction or intra predictionof a video codec, when frequency transformation is performed on theprediction error, a large amount of data may be transformed to 0.According to a video codec, an amount of data may be reduced byreplacing data that is consecutively and repeatedly generated withsmall-sized data.

A multi-layer video codec encodes and decodes a first layer video andone or more second layer videos to remove temporal and spatialredundancies of the first layer video and the second layer videos andredundancy between layers, thereby reducing an amount of data of thefirst layer video and the second layer videos.

SUMMARY

Exemplary embodiments provide inter-layer video encoding methods andapparatuses, and inter-layer video decoding methods, in which a range ofapplication of residual prediction of an image is appropriately limitedto maintain an encoding efficiency and reduce computation complexity.

According to an aspect of an exemplary embodiment, a method andapparatus for inter-layer video encoding, and an inter-layer videodecoding method are provided, in which a range of application ofresidual prediction of a multi-layer image is appropriately limited tomaintain an encoding efficiency and reduce computation complexity.

According to an aspect of an exemplary embodiment, there is provided aninter-layer video decoding method including: obtaining a decoded firstlayer image from a first layer bitstream; determining, based on residualprediction information obtained from a second layer bitstream, whetherto perform residual prediction performed on a second layer block byreferring to a first layer block corresponding to the second layer blockwithin the first layer image; in response to determining not to performthe residual prediction, obtaining luminance compensation informationfrom the second layer bitstream, and determining whether to perform theluminance compensation based on the luminance compensation information;and in response to determining to perform the luminance compensation,performing the luminance compensation on the second layer block.

The inter-layer video decoding method may not perform the luminancecompensation on the second layer block, when it is determined not toperform the luminance compensation.

The inter-layer video decoding method may further include, when it isdetermined to perform the residual prediction, performing the residualprediction to reconstruct the second layer block.

The inter-layer video decoding method may further include, in order toreconstruct the second layer block, a predetermined partition type and apredetermined prediction mode of the second layer block are determined,determining whether to apply the residual prediction based on at leastone of the prediction mode and the partition type of the second layerblock, wherein the determining whether to perform the residualprediction includes, when it is determined to apply the residualprediction, determining whether to perform the residual prediction basedon the residual prediction information.

In the determining of whether to apply the residual prediction, when thedetermined partition type indicates a 2N×2N type, and the determinedprediction mode indicates a merge mode or a skip mode, it may bedetermined to apply the residual prediction.

In the determining of whether to apply the residual prediction, when thedetermined partition type indicates a 2N×2N type, and the determinedprediction mode indicates a merge mode or a skip mode, and luminancecompensation information (ic_flag) obtained from the second layerbitstream indicates 0, it may be determined to apply the residualprediction.

Residual prediction information obtained from the second bitstream mayinclude at least one of flag information, weight information, predictiondirection information, picture type information, and layer information.

The determining of whether to perform the residual prediction mayfurther include: when it is determined to perform the residualprediction, determining a weight value to be applied to residual datawhen performing the residual prediction; and performing the residualprediction by applying the weight value to the residual data.

The weight value may be applied to luma residual data among the residualdata.

According to an aspect of an exemplary embodiment, there is provided aninter-layer video encoding method including: encoding a first layerimage and generating a first layer bitstream including generatedencoding information; determining whether to perform residual predictionperformed on a second layer block by referring to a first layer blockcorresponding to the second layer block within the first layer image,and obtaining residual prediction information; in response todetermining not to perform the residual prediction, determining whetherto perform luminance compensation based on luminance compensationinformation, and obtaining luminance compensation information; and inresponse to determining to perform the luminance compensation,performing the luminance compensation on the second layer block.

The inter-layer video encoding method may further include, when it isdetermined not to perform the luminance compensation, not performing theluminance compensation on the second layer block.

The inter-layer video encoding method may further include, when it isdetermined to perform the residual prediction, performing the residualprediction to encode the second layer block.

The inter-layer video encoding method may further include, in order toencode the second layer block, a predetermined partition type and apredetermined prediction mode of the second layer block are determined,determining whether to apply the residual prediction based on at leastone of the prediction mode and the partition type of the second layerblock, wherein the determining whether to perform the residualprediction includes, when it is determined to apply the residualprediction, determining whether to perform the residual prediction basedon the residual prediction information.

In the determining of whether to apply the residual prediction, when thedetermined partition type indicates a 2N×2N type, and the determinedprediction mode indicates a merge mode or a skip mode, it may bedetermined to apply the residual prediction.

In the determining of whether to apply the residual prediction, when thedetermined partition type indicates a 2N×2N type, and the determinedprediction mode indicates a merge mode or a skip mode, and luminancecompensation information (ic_flag) obtained from the second layerbitstream indicates 0, it may be determined to apply the residualprediction.

The prediction information obtained from the second bitstream mayinclude at least one of flag information, weight information, predictiondirection information, picture type information, and layer information.

The determining of whether to perform the residual prediction mayfurther include: when it is determined to perform the residualprediction, determining a weight value to be applied to residual datawhen performing the residual prediction; and performing the residualprediction by applying the weight value to the residual data.

The weight value may be applied to luma residual data among the residualdata.

According to an aspect of an exemplary embodiment, there is provided aninter-layer video decoding apparatus, including: a first layer decoderconfigured to obtain a decoded first layer image from a first layerbitstream; a residual prediction determiner configured to determine,based on residual prediction information obtained from a second layerbitstream, whether to perform residual prediction performed on a secondlayer block by referring to a first layer block corresponding to thesecond layer block within the first layer image; a luminancecompensation determiner configured to, in response to determining not toperform the residual prediction, obtain luminance compensationinformation from the second layer bitstream and determine whether toperform luminance compensation based on the luminance compensationinformation; and a second layer decoder configured to perform theluminance compensation on the second layer block in response todetermining to perform the luminance compensation.

According to an aspect of an exemplary embodiment, there is provided aninter-layer video encoding apparatus, including: a first layer encoderconfigured to encode a first layer image and generate a first layerbitstream including generated encoding information; a residualprediction determiner configured to determine whether to performresidual prediction performed on a second layer block by referring to afirst layer block corresponding to the second layer block within thefirst layer image, and obtain residual prediction information; aluminance compensation determiner configured to, in response todetermining not to perform the residual prediction, determine whether toperform luminance compensation, and obtain luminance compensationinformation; and a second layer encoder configured to, in response todetermining to perform the luminance compensation, perform the luminancecompensation on the second layer block.

According to an aspect of an exemplary embodiment, an inter-layer videodecoding method includes: obtaining a decoded first layer image from afirst layer bitstream; determining, based on residual predictioninformation obtained from a second layer bitstream, whether to performresidual prediction performed on a second layer block by referring to afirst layer block corresponding to the second layer block within thefirst layer image; when it is determined not to perform the residualprediction, obtaining luminance compensation information from the secondlayer bitstream and determining whether to perform luminancecompensation based on the luminance compensation information; and whenit is determined to perform the luminance compensation, performing theluminance compensation on the second layer block.

The performance of luminance compensation, residual prediction or acombination of these according to various exemplary embodiments isdetermined with respect to only blocks that require the luminancecompensation, residual prediction or a combination of these and more,and luminance compensation, residual prediction or a combination ofthese are not performed on the other blocks so that an increase incomputation load may be reduced and an encoding efficiency may beimproved.

DESCRIPTION OF THE DRAWINGS

These and/or other aspects will become apparent and more readilyappreciated from the following description of the exemplary embodiments,taken in conjunction with the accompanying drawings in which:

FIG. 1A is a block diagram of an inter-layer video encoding apparatus,according to various exemplary embodiments;

FIG. 1B is a flowchart of an inter-layer video encoding method,according to various exemplary embodiments;

FIG. 2A is a block diagram of an inter-layer video decoding apparatus,according to various exemplary embodiments;

FIG. 2B is a flowchart of an inter-layer video decoding method,according to various exemplary embodiments;

FIG. 3 illustrates an inter-layer prediction structure, according to anexemplary embodiment;

FIG. 4 is a flowchart of a method of determining whether to performluminance compensation in an inter-layer video decoding method accordingto an exemplary embodiment;

FIG. 5 illustrates an example of a syntax for determining whether toperform luminance compensation in an inter-layer video decodingapparatus, according to an exemplary embodiment;

FIG. 6 is a flowchart of a method of determining whether to applyresidual prediction in an inter-layer video decoding method according toan exemplary embodiment;

FIG. 7 illustrates an example of syntax for determining whether to applyresidual prediction in an inter-layer video decoding apparatus,according to an exemplary embodiment;

FIG. 8 is a flowchart of a method of determining whether to performresidual prediction in an inter-layer video decoding method according toan exemplary embodiment;

FIG. 9 illustrates an example of syntax for determining whether toperform residual prediction in an inter-layer video decoding apparatus,according to an exemplary embodiment;

FIG. 10 illustrates an example of syntax for applying a weight value toa residual in an inter-layer video decoding apparatus, according to anexemplary embodiment;

FIG. 11 is a block diagram of a video encoding apparatus based on codingunits having a tree structure, according to an exemplary embodiment;

FIG. 12 is a block diagram of a video decoding apparatus based on codingunits having a tree structure, according to an exemplary embodiment;

FIG. 13 is a diagram for describing a concept of coding units accordingto an exemplary embodiment;

FIG. 14 is a block diagram of an image encoder based on coding units,according to an exemplary embodiment;

FIG. 15 is a block diagram of an image decoder based on coding units,according to an exemplary embodiment;

FIG. 16 is a diagram illustrating deeper coding units and partitions,according to an exemplary embodiment;

FIG. 17 is a diagram for describing a relationship between a coding unitand transformation units, according to an exemplary embodiment;

FIG. 18 is a diagram for describing encoding information of codingunits, according to an exemplary embodiment;

FIG. 19 is a diagram of deeper coding units, according to an exemplaryembodiment;

FIGS. 20 through 22 are diagrams for describing a relationship betweencoding units, prediction units, and transformation units, according toan exemplary embodiment;

FIG. 23 is a diagram for describing a relationship between a codingunit, a prediction unit, and a transformation unit, according toencoding mode information of Table 1;

FIG. 24 is a diagram of a physical structure of a disc in which aprogram is stored, according to an exemplary embodiment;

FIG. 25 is a diagram of a disc drive for recording and reading a programby using a disc;

FIG. 26 is a diagram of an overall structure of a content supply systemfor providing a content distribution service;

FIGS. 27 and 28 are diagrams respectively of an external structure andan internal structure of a mobile phone to which a video encoding methodand a video decoding method are applied, according to an exemplaryembodiment;

FIG. 29 is a diagram of a digital broadcast system to which acommunication system is applied, according to an exemplary embodiment;and

FIG. 30 is a diagram illustrating a network structure of a cloudcomputing system using a video encoding apparatus and a video decodingapparatus, according to an exemplary embodiment.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENT

Hereinafter, an inter-layer video encoding method and an inter-layervideo decoding method in which whether to perform luminancecompensation, whether to apply residual prediction, whether to performresidual prediction, or whether to perform a combination of two or morethereof is determined according to various exemplary embodiments will bedescribed with reference to FIGS. 1A through 10. A video encoding methodand a video decoding method, based on coding units having a treestructure according to various exemplary embodiments that are applicableto the inter-layer video encoding method and the inter-layer videodecoding method will be described with reference to FIGS. 11 through 23.In addition, various exemplary embodiments to which the video encodingmethod and the video decoding method will be described with reference toFIGS. 24 through 30.

Hereinafter, an ‘image’ may denote a still image or a moving image of avideo, or a video itself.

Hereinafter, a ‘sample’ that is data allocated to a sampling location ofan image may mean data that is a processing target. For example, pixelsor residuals in an image of a spatial area may be samples.

An inter-layer video encoding apparatus and method and an inter-layervideo decoding apparatus and method according to an exemplary embodimentwill now be described with reference to FIGS. 1A through 10.

FIG. 1A is a block diagram of an inter-layer video encoding apparatus 10according to various exemplary embodiments. FIG. 1B is a flowchart of aninter-layer video encoding method according to various exemplaryembodiments.

The inter-layer video encoding apparatus 10 according to variousexemplary embodiments may include a first layer encoder 12, a residualprediction determiner 14, a luminance compensation determiner 16, and asecond layer encoder 18. The residual prediction determiner 14 and theluminance compensation determiner 16 may be included in the second layerencoder 18. Also, the residual prediction determiner 14 and theluminance compensation determiner 16 may be located outside the secondlayer encoder 18.

The inter-layer video encoding apparatus 10 according to variousexemplary embodiments may classify and encode a plurality of imagesequences for each layer according to scalable video coding and mayoutput a separate stream including data encoded for each layer. Theinter-layer video encoding apparatus 10 may encode first layer imagesequences and second layer image sequences according to differentlayers.

The first layer encoder 12 may encode first layer images and output afirst layer bitstream including encoding data of the first layer images.

The second layer encoder 18 may encode second layer images and output asecond layer bitstream including encoding data of the second layerimages.

The first layer bitstream and the second layer bitstream may bemultiplexed to be output as a single bitstream.

For example, according to scalable video coding based on spatialscalability, low resolution images may be encoded as the first layerimages, and high resolution images may be encoded as the second layerimages. An encoding result of the first layer images may be output in afirst layer bitstream. An encoding result of the second layer images maybe output in a second layer bitstream.

As another example, a multi-view video may be encoded according toscalable video coding. Left view images may be encoded as first layerimages, and right view images may be encoded as second layer images.Alternatively, center view images, left view images, and right viewimages may be respectively encoded, and here, the center view images maybe encoded as first layer images, and the left view images may beencoded as the first layer images, and the right view images may beencoded as second layer images.

As another example, temporal hierarchical prediction (temporal level)may be classified based on temporal scalability, and temporal layers maybe respectively encoded in layers. A second layer bitstream includingencoding information of a high speed frame rate may be output by furtherencoding images of the high frame rate with reference to images of thebasic frame rate.

Scalable video coding may be performed on a first layer and a pluralityof second layers. In the presence of three or more second layers, firstlayer images, first second layer images, second second layers images, .. . , Kth second layer images may be encoded. Accordingly, an encodingresult of the first layer images may be output in a first layerbitstream, and encoding results of the first second layer images, secondsecond layers images, . . . , Kth second layer images may berespectively output in first, second, . . . Kth second layer bitstreams.

The inter-layer video encoding apparatus 10 according to variousexemplary embodiments may perform inter prediction for predicting acurrent image by referring to images of a single layer. A motion vectorindicating motion information between the current image and a referenceimage and a residual between the current image and the reference imagemay be generated through inter prediction.

The inter-layer video encoding apparatus 10 may perform inter-layerprediction for predicting second layer images by referring to the firstlayer images.

When the inter-layer video encoding apparatus 10 according to anexemplary embodiment permits three or more layers such as a first layer,a second layer, a third layer, etc., the inter-layer video encodingapparatus 10 may perform inter-layer prediction between a first layerimage and a third layer image and inter-layer prediction between asecond layer image and the third layer image according to a multi-layerprediction structure.

A position differential component between the current image and areference image of a different layer and a residual between the currentimage and the reference image of the different layer may be generatedthrough inter-layer prediction.

An inter-layer prediction structure will be described in detail withreference to FIG. 3 later.

The inter-layer video encoding apparatus 10 according to variousexemplary embodiments encodes each video image for each respective blockaccording to each layer. A block may have a square shape, a rectangularshape, or any geometric shape and is not limited to a data unit having apredetermined size. A block may be a maximum coding unit, a coding unit,a prediction unit, a transformation unit, or the like from among codingunits having a tree structure. The maximum encoding unit includingcoding units having the tree structure is diversely referred to as acoding block tree, a block tree, a root block tree, a coding tree, acoding root or a tree trunk. Video encoding and decoding methods basedon coding units having the tree structure will be described later withreference to FIGS. 11 through 23.

Inter prediction and inter-layer prediction may be performed based on adata unit of the coding unit, the prediction unit, or the transformationunit.

The first layer encoder 12 according to various exemplary embodimentsmay perform source coding operations including inter prediction or intraprediction on the first layer images to generate symbol data. The symboldata represents a sample value of each coding parameter and a samplevalue of the residual.

For example, the first layer encoder 12 may perform inter prediction, orintra prediction, transformation and quantization on samples in a dataunit of the first layer images, generate symbol data, perform entropyencoding on the symbol data, and generate a first layer bitstream.

The second layer encoder 18 may encode the second layer images based onthe coding units having the tree structure. The second layer encoder 18may perform inter/intra prediction, transformation and quantization onsamples in a coding unit of the second layer images, generate symboldata, perform entropy encoding on the symbol data, and generate ansecond layer bitstream.

The second layer encoder 18 according to various exemplary embodimentsmay perform inter-layer prediction that predicts a second layer image byusing a reconstruction sample of a first layer image. The second layerencoder 18 may generate a second layer prediction image by using a firstlayer reconstruction image, and encode a prediction error between thesecond layer original image and the second layer prediction image inorder to encode the second layer original image among the second layerimage sequences through the inter-layer prediction structure.

The second layer encoder 18 may perform inter-layer prediction on thesecond layer image for each block such as the coding unit or theprediction unit and determine a block of the first layer image to whicha block of the second layer image is to refer. For example, areconstruction block of the first layer image positioned incorrespondence to a position of a current block in the second layerimage may be determined. The second layer encoder 18 may determine asecond layer prediction block by using the first layer reconstructionblock corresponding to the second layer block.

The second layer encoder 18 may use the second layer prediction blockdetermined by using the first layer reconstruction block according to aninter-layer prediction structure, as a reference image for inter-layerprediction of a second layer original block. The second layer encoder 18may perform entropy encoding on an error between a sample value of thesecond layer prediction block and a sample value of the second layeroriginal block, i.e., a residual according to inter-layer prediction, byusing the first layer reconstruction image.

As described above, the second layer encoder may encode a current layerimage sequence by referring to first layer reconstruction images throughthe inter-layer prediction structure. However, the second layer encoder18 according to various exemplary embodiments may encode the secondlayer image sequence according to a single layer prediction structurewithout referring to different layer samples. Thus, it is not limited toconstrue that the second layer encoder 18 performs only inter-layerprediction in order to encode the second layer image sequence.

As described above, when the inter-layer video encoding apparatus 10encodes a multi-view video, the first layer encoder may encode a firstview video, and the second layer encoder 18 may encode a second viewvideo. Video for each view may be captured by different cameras or maybe acquired using different lenses. Since characteristics of a capturingangle, illumination, or an imaging tool (a camera, a lens, etc.) foreach view may be different, a phenomenon may occur whereby luminance isnot identical between videos acquired for each view. Such a luminancemismatch phenomenon may be related to a difference in a sample valuebetween videos for each view.

If luminance is not identical between videos for each view, since theamount of inter-layer prediction errors further increases, encodingefficiency may be reduced. Accordingly, considering luminance mismatchbetween views, the luminance compensation determiner 16 of theinter-layer video encoding apparatus 10 may compensate for and encode aluminance difference of video for each view. For example, a luminancedifference between a first view image encoded by the first layer encoder12 and a second view image encoded by the second layer encoder 18 may beencoded. Since the luminance difference of the second view image withrespect to the first view image is encoded, luminance may be compensatedfor when the second layer encoder 18 encodes a second view video.

Residual prediction may be performed between layers in the inter-layerprediction structure. For example, when encoding a second layer image,the second layer encoder 18 may acquire a residual of a second layerblock by referring to a residual a first layer block corresponding tothe second layer block within a first image.

When performing temporal prediction on a second view image, the secondlayer encoder 18 may determine a first layer reference block byapplying, to the first layer block pointed by a disparity vector (DV) ofthe second layer block, a motion vector indicating motion informationbetween the second layer block and a second layer reference blockreferred to by the second layer block.

The second layer encoder 18 may acquire a residual between the firstlayer block and the first layer reference block referred to by the firstlayer block. The second layer encoder 18 may acquire a prediction blockimage of the second layer block by adding the acquired first layerresidual to the second layer reference block, and acquire a residual ofthe second layer block by using the second layer prediction block image.A residual may be predicted using other various methods than theabove-described method, and the method of predicting a residual is notlimited thereto.

Residual prediction between layers, luminance compensation, or acombination of these may be performed in an inter-layer predictionstructure, and thus, an amount of arithmetic operations may beincreased. Accordingly, the luminance compensation determiner 16 maydetermine whether to perform luminance compensation based on luminancecompensation information. The residual prediction determiner 14 maydetermine whether to perform residual prediction based on residualprediction information.

A detailed operation of the inter-layer video encoding apparatus 10 thatconsiders luminance compensation and residual prediction will bedescribed with reference to FIG. 1B below.

FIG. 1B is a flowchart of an inter-layer video encoding method,according to various exemplary embodiments.

In operation S11, the first layer encoder 12 may encode a first layerimage and generate a first layer bitstream including generated encodinginformation.

In operation S13, the residual prediction determiner 14 may determine,based on residual prediction information, whether to perform residualprediction performed on a second layer block by referring to a firstlayer block corresponding to the second layer block within the firstlayer image.

According to an exemplary embodiment of the inventive concept, theresidual prediction determiner 14 may acquire residual predictioninformation. The residual prediction determiner 14 may indicate whetherto perform residual prediction by using the residual predictioninformation. The residual prediction information may include flaginformation, weight information, prediction direction information,picture type information, layer information, or a combination of atleast two of these.

According to an exemplary embodiment of the inventive concept, theresidual prediction determiner 14 may indicate whether to performresidual prediction performed on the second layer block, by using flaginformation included in residual prediction information. For example,when the residual prediction determiner 14 has determined to performresidual prediction, flag information may indicate that residualprediction is performed.

According to an exemplary embodiment of the inventive concept, theresidual prediction determiner 14 may indicate whether residualprediction performed on the second layer block is performed, by usingweight information included in residual prediction information. Forexample, when the residual prediction determiner 14 has determined notto perform residual prediction, weight information may include ‘0.’ Whenthe residual prediction determiner 14 has determined to perform residualprediction, weight information may include a value other than ‘0.’

According to an exemplary embodiment of the inventive concept, theresidual prediction determiner 14 may determine whether to performresidual by using prediction direction information. For example, ifprediction direction information allows only temporal prediction, and ifcurrent prediction is view direction prediction, the residual predictiondeterminer 14 may determine not to perform residual prediction.

According to an exemplary embodiment of the inventive concept, theresidual prediction determiner 14 may indicate whether residualprediction performed on the second layer block is performed, based onflag information, weight information, prediction direction information,picture type information, layer information, or a combination of atleast two of these included in residual prediction information. Forexample, even when flag information included in residual predictioninformation indicates that residual prediction is performed, weightinformation included in the residual prediction information may be setto include 0 so as to indicate that residual prediction is notperformed. The residual prediction determiner 14 may also indicatewhether residual prediction is performed, by using other variouscombinations of pieces of information included in residual predictioninformation, and the method of indicating whether residual prediction isperformed is not limited thereto.

According to an exemplary embodiment of the inventive concept, when theresidual prediction determiner 14 has determined to perform residualprediction, the residual prediction determiner 14 may determine a weightvalue to be applied to a residual. For example, the residual predictiondeterminer 14 may determine to perform residual prediction, anddetermine 1 as a weight value to be applied to a residual duringresidual prediction. Also, ½ may be determined as a weight value to beapplied to a residual during residual prediction.

According to an exemplary embodiment of the inventive concept, theresidual prediction determiner 14 may determine whether residualprediction is applicable, before determining whether to perform residualprediction. Also, whether residual prediction is applicable may beindicated by using residual prediction information. For example, whenthe residual prediction determiner 14 has determined that residualprediction is applicable, ‘rpEnableFlag’ may include 1.

When the residual prediction determiner 14 has determined that residualprediction is not applicable, ‘rpEnableFlag’ may indicate 0. When theresidual prediction determiner 14 has determined that residualprediction is not applicable, the residual prediction determiner 14 mayomit an operation of determining whether to perform residual prediction.

According to an exemplary embodiment of the inventive concept, apartition type of the second layer block may have a size such as 2N×2N,2N×N, N×2N, or N×N. 2N×2N indicates a partition in the form of a codingunit that is not split. 2N×N and N×2N indicate a coding unit that isdivided by half only in one of a height direction or a width directionto be split into two partitions. N×N indicates a coding unit divided byhalf both in a height direction and in a width direction to be splitinto four partitions. Examples of a partition type include symmetricalpartitions that are obtained by symmetrically splitting a height orwidth of the prediction unit, partitions obtained by asymmetricallysplitting the height or width of the prediction unit, such as 1:n orn:1, partitions that are obtained by geometrically splitting theprediction unit, and partitions having arbitrary shapes.

According to an exemplary embodiment of the inventive concept, when aprediction mode of the second layer block is a Merge mode or a SKIPmode, and a partition type of the second layer block is 2N×2N, theresidual prediction determiner 24 may determine that residual predictionis applicable to the second layer block.

When a prediction mode of the second layer block is a Merge mode or aSKIP mode, and a partition type of the second layer block is 2N×2N, andluminance compensation information (ic_flag) indicates 0, the residualprediction determiner 24 may determine that residual prediction isapplicable to the second layer block.

In operation S15, when the residual prediction determiner 14 hasdetermined not to perform residual prediction, the luminancecompensation determiner 16 may determine whether to perform luminancecompensation, and may indicate whether to perform luminance compensationby using luminance compensation information.

According to an exemplary embodiment of the inventive concept, theluminance compensation determiner 16 may determine whether to performluminance compensation performed on the second layer block. Also, theluminance compensation determiner 16 may indicate, by using luminancecompensation information, whether luminance compensation is performed.For example, luminance compensation information may include “ic_flag.”When the luminance compensation determiner 16 determines not to performluminance compensation, the luminance compensation determiner 16 may set“ic_flag” to 0. Also, when the luminance compensation determiner 16determines to perform luminance compensation, the luminance compensationdeterminer 16 may set “ic_flag” to 1.

According to an exemplary embodiment of the inventive concept, when theresidual prediction determiner 14 has determined to perform residualprediction on the second layer block, the luminance compensationdeterminer 16 may omit an operation of determining luminancecompensation information. For example, when the residual predictiondeterminer 14 determines to perform residual prediction based onresidual prediction information, the luminance compensation determiner16 may omit an operation of encoding ‘ic_flag.’

In operation S17, when the luminance compensation determiner 16 hasdetermined to perform luminance compensation, the second layer encoder18 may perform luminance compensation on the second layer block.

According to an exemplary embodiment of the inventive concept, when theluminance compensation determiner 16 has determined to perform luminancecompensation, the second layer encoder 18 may perform luminancecompensation on the second layer block. For example, when the residualprediction determiner 14 determines not to perform residual prediction,and the luminance compensation determiner 16 determines to performluminance compensation, the second layer encoder 18 may performluminance compensation on the second layer block to encode the secondlayer block, and may generate a second layer bitstream by using theencoded second layer block.

According to an exemplary embodiment of the inventive concept, when theresidual prediction determiner 14 determines to perform residualprediction, the second layer encoder 18 may omit determination of theluminance compensation determiner 16 and perform residual prediction toencode the second layer block. For example, when the residual predictiondeterminer 14 determines to perform residual prediction, the secondlayer encoder 18 may omit determination and luminance compensation ofthe luminance compensation determiner 16, and may perform residualprediction to encode the second layer block.

According to an exemplary embodiment of the inventive concept, when theresidual prediction determiner 14 determines to perform residualprediction, the second layer encoder 18 may apply to a residual a weightvalue determined by the residual prediction determiner 14. For example,when a weight value determined by the residual prediction determiner 14is 1, the second layer encoder 18 may use an acquired residual withoutany change to encode the second layer block. When a weight valuedetermined by the residual prediction determiner 14 is ½, the secondlayer encoder 18 may apply the weight value of ½ to the acquiredresidual component to encode the second layer block.

According to an exemplary embodiment of the inventive concept, whenapplying a weight value to a residual, the second layer encoder 18 mayapply different weight values according to respective color elements.For example, when a weight value determined by the second layer encoder18 is ½, the second layer encoder 12 may apply the weight value of ½ toa luma residual, and may apply no weight value to a chroma residual.

According to an exemplary embodiment of the inventive concept, in aMerge mode or a Skip mode, residual prediction information may includean arbitrary value regardless of a prediction direction. For example,even when the residual encoder 12 does not perform residual prediction,by considering an entropy coding efficiency, residual predictioninformation may have another value different from a default value. Indetail, when residual prediction is not performed,‘iv_res_pred_weight_idx’ may have a value other than ‘0.’

The inter-layer video encoding apparatus 10 according to variousexemplary embodiments of the inventive concept may include a centralprocessor (not shown) that generally controls the first layer encoder12, the residual prediction determiner 14, the luminance compensationdeterminer 16, and the second layer encoder 18. Alternatively, the firstlayer encoder 12, the residual prediction determiner 14, the luminancecompensation determiner 16, and the second layer encoder 18 may operateby their respective processors (not shown), and the inter-layer videoencoding apparatus 10 may generally operate according to interactions ofthe processors (not shown). Alternatively, the first layer encoder 12,the residual prediction determiner 14, the luminance compensationdeterminer 16, and the second layer encoder 18 may be controlledaccording to the control of an external processor (not shown) of theinter-layer video encoding apparatus 10.

The inter-layer video encoding apparatus 10 may include one or more datastorage units (not shown) in which input and output data of the firstlayer encoder 12, the residual prediction determiner 14, the luminancecompensation determiner 16, and the second layer encoder 18 is stored.The inter-layer video encoding apparatus 10 may include a memory controlunit (not shown) that observes data input and output of the data storageunits (not shown).

The inter-layer video encoding apparatus 10 may operate in connectionwith an internal video encoding processor or an external video encodingprocessor so as to output video encoding results, thereby performing avideo encoding operation including transformation. The internal videoencoding processor of the inter-layer video encoding apparatus 10 mayimplement a video encoding operation as a separate processor. Also, abasic video encoding operation may be implemented as the inter-layervideo encoding apparatus 10 or a central processor or a graphicprocessor includes a video encoding processing module.

FIG. 2A is a block diagram of an inter-layer video decoding apparatus 20according to various exemplary embodiments.

The inter-layer video decoding apparatus 20 according to variousexemplary embodiments may include a first layer encoder 22, a residualprediction determiner 24, a luminance compensation determiner 26, and asecond layer decoder 28. The residual prediction determiner 24 and theluminance compensation determiner 26 may be included in the second layerdecoder 28. The luminance compensation determiner 26 according toanother exemplary embodiment may be located outside the second layerdecoder 28.

The inter-layer video decoding apparatus 20 according to variousexemplary embodiments may receive bitstreams for each layer according toscalable encoding. The number of layers of the bitstreams received bythe inter-layer video decoding apparatus 20 is not limited. However, forconvenience of description, an exemplary embodiment in which the firstlayer encoder 22 of the inter-layer video decoding apparatus 20 receivesand decodes a first layer bitstream and the second layer decoder 28receives and decodes a second layer bitstream will be described indetail.

For example, the inter-layer video decoding apparatus 20 based onspatial scalability may receive streams in which image sequences ofdifferent resolutions are encoded according to different layers. A lowresolution image sequence may be reconstructed by decoding the firstlayer bitstream, and a high resolution image sequence may bereconstructed by decoding the second layer bitstream.

As another example, a multi-view video may be decoded according toscalable video coding. When a stereoscopic video stream is received inmultiple layers, the first layer bitstream may be decoded to reconstructleft view images. The second layer bitstream may be further decoded tothe first layer bitstream to reconstruct right view images.

Alternatively, when a multi-view video stream is received in multiplelayers, the first layer bitstream may be decoded to reconstruct centerview images. The second layer bitstream may be further decoded to thefirst layer bitstream to reconstruct the left view images. A third layerbitstream may be further decoded to the first layer bitstream toreconstruct the right view images.

As another example, scalable video coding based on scalability may beperformed. The first layer bitstream may be decoded to reconstruct baseframe rate images. The second layer bitstream may be further decoded tothe first layer bitstream to reconstruct high speed frame rate images.

In the presence of three or more second layers, first layer images maybe reconstructed from the first layer bitstream. If the second layerbitstream is further decoded by referring to the first layerreconstruction images, second layer images may be further reconstructed.If a Kth layer bitstream is further decoded by referring to the secondlayer reconstruction images, Kth layer images may be furtherreconstructed.

The inter-layer video decoding apparatus 20 may obtain encoded data ofthe first layer images and second layer images from the first layerbitstream and the second layer bitstream and may further obtain a motionvector generated through inter prediction and prediction informationgenerated through inter-layer prediction.

For example, the inter-layer video decoding apparatus 20 may decodeinter-predicted data for each layer and may decode inter-layer-predicteddata between a plurality of layers. Reconstruction may be performedthrough motion compensation and inter-layer decoding based on a codingunit or a prediction unit.

Motion compensation for a current image is performed by referring toreconstruction images predicted through inter prediction of a same layeron each layer stream, and thus images may be reconstructed. Motioncompensation means an operation of synthesizing a reference imagedetermined by using a motion vector of the current image and a residualof the current image and reconfiguring a reconstruction image of thecurrent image.

The inter-layer video decoding apparatus 20 may perform inter-layerdecoding with reference to the first layer images so as to decode asecond layer image predicted through inter-layer prediction. Inter-layerdecoding means an operation of reconfiguring a reconstruction image of acurrent image by synthesizing a reference image of a different layerdetermined to predict the current image and a residual of the currentimage.

The inter-layer video decoding apparatus 20 according to an exemplaryembodiment may perform inter-layer decoding for reconstructing the thirdlayer images predicted with reference to the second layer images. Aninter-layer prediction structure will be described in detail later withreference to FIG. 3.

However, the second layer decoder 28 according to various exemplaryembodiments may decode the second layer bitstream without referring tothe first layer image sequence. Thus, it is not limited to construe thatthe second layer decoder 28 performs only inter-layer prediction inorder to decode the second layer image sequence.

The inter-layer video decoding apparatus 20 decodes each image of avideo for each block. A block may include a maximum encoding unit, acoding unit, a prediction unit, a transformation unit, etc. among codingunits having a tree structure.

The first layer encoder 22 may decode the first layer image by usingencoding symbols of a parsed first layer image. If the inter-layer videodecoding apparatus 20 receives encoded streams based on coding unitshaving a tree structure, the first layer encoder 22 may perform decodingbased on the coding units according to the tree structure for eachmaximum coding unit of the first layer bitstream.

The first layer encoder 22 may perform entropy encoding for each maximumcoding unit and may obtain encoding information and encoded data. Thefirst layer encoder 22 may perform inverse quantization and inversetransformation on the encoded data obtained from streams to reconstructa residual. The first layer encoder 22 according to another exemplaryembodiment may directly receive a bitstream of quantized transformationcoefficients. A residual of the images may be reconstructed as a resultof performing inverse quantization and inverse transformation on thequantized transformation coefficients.

The first layer encoder 22 may reconstruct the first layer images bysynthesizing a prediction image and the residual through motioncompensation between same layer images.

The second layer decoder 28 may generate a second layer prediction imageby using samples of a first layer reconstruction image according to theinter-layer prediction structure. The second layer decoder 28 may decodethe second layer bitstream to obtain a prediction error according tointer-layer prediction. The second layer decoder 28 may combine thesecond layer prediction image and the prediction error, therebygenerating the second layer reconstruction image.

The second layer decoder 28 may determine the second layer predictionimage by using the first layer reconstruction image decoded by the firstlayer encoder 22. The second layer decoder 28 may determine a block ofthe first layer image to which a block such as a coding unit or aprediction unit of the second layer image is to refer according to theinter-layer prediction structure. For example, a reconstruction block ofthe first layer image located in the second layer image incorrespondence to a location of a current block may be determined. Thesecond layer decoder 28 may determine a second layer prediction blockusing a first layer reconstruction block corresponding to a second layerblock.

The second layer decoder 28 may use the second layer prediction blockdetermined using the first layer reconstruction block according to theinter-layer prediction structure as a reference image for inter-layerpredicting a second layer original block. In this case, the second layerdecoder 28 may reconstruct the second layer block by synthesizing asample value of the second layer prediction block determined using thefirst layer reconstruction image and a residual according to inter-layerprediction.

According to spatial scalable video coding, when the first layer decoder22 reconstructs the first layer image of a different resolution fromthat of the second layer image, the second layer decoder 28 mayinterpolate the first layer reconstruction image to resize the firstlayer reconstruction image to have the same resolution as that of thesecond layer original image. The interpolated first layer reconstructionimage may be determined as the second layer prediction image forinter-layer prediction.

Therefore, the first layer decoder 22 of the inter-layer video decodingapparatus 20 may reconstruct the first layer image sequence by decodingthe first layer bitstream, and the second layer decoder 28 mayreconstruct the second layer image sequence by decoding the second layerbitstream.

In consideration of a luminance mismatch between views, the luminancecompensation determiner 26 of the inter-layer video decoding apparatus20 may compensate for and reconstruct a luminance difference betweenvideos for each view. For example, a luminance difference between afirst view image decoded by the first layer decoder 22 and a second viewimage decoded by the second layer decoder 28 may be acquired from abitstream. Since the luminance difference between the second view imageand the first view image is acquired, it may be determined whether toperform luminance compensation when the second layer decoder 28 decodesa second view video.

The residual prediction determiner 24 of the inter-layer video decodingapparatus 20 may determine whether to perform residual prediction, andthe second layer decoder 28 may perform residual prediction toreconstruct the second layer block. For example, the residual predictiondeterminer 24 may determine whether to perform residual prediction basedon residual prediction information acquired from the second bitstream,and when it is determined to perform residual prediction on the secondlayer block, the second layer decoder 28 may perform residual predictionby using a residual acquired from the second bitstream.

A detailed operation of the inter-layer video decoding apparatus 20capable of performing residual prediction and luminance compensationwill now be described with reference to FIG. 2B.

FIG. 2B is a flowchart of an inter-layer video decoding method accordingto various exemplary embodiments.

In operation S21, the first layer decoder 22 may reconstruct a firstlayer image based on encoding information acquired from a first layerbitstream.

In operation S23, the residual prediction determiner 24 may determine,based on residual prediction information, whether to perform residualprediction performed on a second layer block by referring to a firstlayer block corresponding to the second layer block within the firstlayer image.

According to an exemplary embodiment of the inventive concept, theresidual prediction determiner 24 may determine, by using flaginformation included in residual prediction information, whether toperform residual prediction performed on the second layer block. Forexample, when flag information indicates that residual prediction isperformed, the residual prediction determiner 24 may determine toperform residual prediction.

According to an exemplary embodiment of the inventive concept, theresidual prediction determiner 24 may determine, by using weightinformation included in residual prediction information, whether toperform residual prediction performed on the second layer block. Forexample, when weight information indicates 0, the residual predictiondeterminer 24 may determine not to perform residual prediction. Also,when weight information indicates a value that is not ‘0,’ the residualprediction determiner 24 may determine to perform residual prediction.

According to an exemplary embodiment of the inventive concept, theresidual prediction determiner 24 may determine, by using predictiondirection information included in residual prediction information,whether to perform residual prediction performed on the second layerblock. For example, if prediction direction information allows onlytemporal prediction, and if current prediction is view directionprediction, the residual prediction determiner 24 may determine not toperform residual prediction.

According to an exemplary embodiment of the inventive concept, theresidual prediction determiner 24 may determine whether to performresidual prediction performed on the second layer block, based on flaginformation, weight information, prediction direction information,picture type information, layer information included in residualprediction information or a combination of at least two thereof. Forexample, even when flag information included in residual predictioninformation indicates that residual prediction is performed, if weightinformation included in the residual prediction information indicates 0,the residual prediction determiner 24 may determine not to performresidual prediction. In addition, even when weight information includes1 indicating that residual prediction is performed, if predictiondirection information allows only temporal prediction but does not allowview direction prediction, the residual prediction determiner 24 maydetermine not to perform residual prediction. The residual predictiondeterminer 24 may determine whether to perform residual prediction byusing various combinations of pieces of information included in residualprediction information, other than the methods described above, and themethod of determining whether to perform residual prediction is notlimited thereto.

According to an exemplary embodiment of the inventive concept, when theresidual prediction determiner 24 has determined to perform residualprediction, the residual prediction determiner 24 may determine a weightvalue to be applied to a residual. For example, when the residualprediction determiner 24 has determined to perform residual prediction,and weight information included in residual prediction informationacquired by the residual prediction determiner 24 indicates 1, theresidual prediction determiner 24 may determine 1 as a weight value tobe applied to a residual during residual prediction. Also, when weightinformation indicates 2, the residual prediction determiner 24 maydetermine ½ as a weight value to be applied to a residual duringresidual prediction.

According to an exemplary embodiment of the inventive concept, beforedetermining whether residual prediction may be performed or not, theresidual prediction determiner 24 may determine whether residualprediction is applicable. For example, when ‘rpEnableFlag’ obtained fromthe second layer bitstream indicates 1, the residual predictiondeterminer 24 may determine that residual prediction is applicable, anddetermine whether to perform residual prediction. Also, when‘rpEnableFlag’ obtained from the second layer bitstream indicates 0, thesecond layer decoder 28 may determine that residual prediction is notapplicable, and may omit an operation of determining whether to performresidual prediction, and may determine whether to perform luminancecompensation.

When a prediction mode of the second layer block is a Merge mode or aSKIP mode, and a partition type of the second layer block is 2N×2N, theresidual prediction determiner 24 may determine that residual predictionis applicable to the second layer block.

In operation S25, when the residual prediction determiner 24 hasdetermined not to perform residual prediction, the luminancecompensation determiner 26 may acquire luminance compensationinformation, and may determine whether to perform luminance compensationbased on the luminance compensation information.

According to an exemplary embodiment of the inventive concept, theluminance compensation determiner 26 may determine whether to performluminance compensation performed on the second layer block, by usingluminance compensation information. Luminance compensation informationis defined as information referred to by the luminance compensationdeterminer 26 to determine whether to perform luminance compensation.For example, luminance compensation may include “ic_flag.” When“ic_flag” obtained by the luminance compensation determiner 26 indicates0, the luminance compensation determiner 26 may determine not to performluminance compensation. Also, when “ic_flag” obtained by the luminancecompensation determiner 26 indicates 1, the luminance compensationdeterminer 26 may determine to perform luminance compensation.

According to an exemplary embodiment of the inventive concept, when theresidual prediction determiner 24 has determined to perform residualprediction on the second layer block, the luminance compensationdeterminer 26 may omit an operation of obtaining luminance compensationinformation. For example, when the residual prediction determiner 24determines, based on residual prediction information, to performresidual prediction, the luminance compensation determiner 26 may omitan operation of obtaining luminance compensation information.

In operation S27, when the luminance compensation determiner 26 hasdetermined to perform luminance compensation, the second layer decoder28 may perform luminance compensation on the second layer block.

According to an exemplary embodiment of the inventive concept, when theluminance compensation determiner 26 has determined to perform luminancecompensation, the second layer decoder 28 may perform luminancecompensation on the second layer block. For example, when the residualprediction determiner 24 determines not to perform residual prediction,and the luminance compensation determiner 26 determines to performluminance compensation, the second layer decoder 28 may performluminance compensation on the second layer block to decode the secondlayer block, and may obtain a second layer image by using the decodedsecond layer block.

According to an exemplary embodiment of the inventive concept, when theresidual prediction determiner 24 determines to perform residualprediction, the second layer decoder 28 may omit determination of theluminance compensation determiner 26 and perform residual prediction todecode the second layer block. For example, when the residual predictiondeterminer 24 determines to perform residual prediction, the secondlayer decoder 28 may omit determination and luminance compensation ofthe luminance compensation determiner 26, and may perform residualprediction to decode the second layer block.

According to an exemplary embodiment of the inventive concept, when theresidual prediction determiner 24 determines to perform residualprediction, the second layer decoder 28 may apply a weight valuedetermined by the residual prediction determiner 24 to a residual duringresidual prediction. For example, when a weight value determined by theresidual prediction determiner 24 is 1, the second layer decoder 28 mayuse an obtained residual without any change to decode the second layerblock. When a weight value determined by the residual predictiondeterminer 24 is ½, the second layer decoder 28 may apply the weightvalue of ½ to the obtained residual to decode the second layer block.

According to an exemplary embodiment of the inventive concept, whenapplying a weight value to a residual, the second layer decoder 28 mayapply different weight values according respective color elements. Forexample, when a weight value determined by the second layer decoder 28is ½, the second layer decoder 28 may apply the weight value of ½ to aluma residual, and may apply no weight value to a chroma residual.

The inter-layer video decoding apparatus 20 according to variousexemplary embodiments may include a central processor (not shown) thatgenerally controls the first layer decoder 22, the residual predictiondeterminer 24, the luminance compensation determiner 26, and the secondlayer decoder 28. Alternatively, the first layer decoder 22, theresidual prediction determiner 24, the luminance compensation determiner26, and the second layer decoder 28 may operate by their respectiveprocessors (not shown), and the inter-layer video decoding apparatus 20may generally operate according to interactions of the processors (notshown). Alternatively, the first layer decoder 22, the residualprediction determiner 24, the luminance compensation determiner 26, andthe second layer decoder 28 may be controlled according to the controlof an external processor (not shown) of the inter-layer video decodingapparatus 20 according to various exemplary embodiments.

The inter-layer video decoding apparatus 20 according to variousexemplary embodiments may include one or more data storage units (notshown) in which input and output data of the first layer decoder 22, theresidual prediction determiner 24, the luminance compensation determiner26, and the second layer decoder 28 is stored. The inter-layer videodecoding apparatus 20 may include a memory control unit (not shown) thatobserves data input and output of the data storage units (not shown).

The inter-layer video decoding apparatus 20 according to variousexemplary embodiments may operate in connection with an internal videodecoding processor or an external video decoding processor so as toreconstruct a video through video decoding, thereby performing a videodecoding operation including inverse transformation. The internal videodecoding processor of the inter-layer video decoding apparatus 20 mayimplement a basic video decoding operation as not only a separateprocessor but also the inter-layer video decoding apparatus 20 or acentral processor or a graphic processor includes a video decodingprocessing module.

FIG. 3 illustrates an inter-layer prediction structure according tovarious exemplary embodiments.

The inter-layer video encoding apparatus 10 according to variousexemplary embodiments may prediction encode base view images, left viewimages, and right view images according to a reproduction order 30 of amulti-view video prediction structure shown in FIG. 3.

According to the reproduction order 30 of the multi-view videoprediction structure of the related art, images of the same view may bearranged in a horizontal direction. Thus, left view images “Left” may bearranged in a line in the horizontal direction, base view images“Center” may be arranged in a line in the horizontal direction, andright view images “Right” may be arranged in a line in the horizontaldirection. The base view images may be center view images compared tothe left and right view images.

Images having the same POC order may be arranged in a verticaldirection. A POC of images indicates a reproduction order of imagesconstituting video. “POC X” in the multi-view video prediction structure30 indicates a relative reproduction order of images positioned in acorresponding column. The smaller the number of X, the earlier thereproduction order, and the greater the number of X, the later thereproduction order.

Therefore, according to the reproduction order 30 of the multi-viewvideo prediction structure of the related art, the left view images“Left” may be arranged in the horizontal direction according to the POC(reproduction order), the base view images “Center” may be arranged inthe horizontal direction according to the POC (reproduction order), andthe right view images “Right” may be arranged in the horizontaldirection according to the POC (reproduction order). Also, the left andright view images positioned in the same column as that of the base viewimages have different views but have the same POC (reproduction order).

Four consecutive images of view images constitute a single GOP (Group ofPicture). Each GOP includes images between consecutive anchor picturesand a single anchor picture (Key Picture).

An anchor picture is a random access point. In this regard, when apredetermined reproduction position is selected from images that arearranged according to a reproduction order of video, that is, accordingto a POC, an anchor picture of which a POC is closest to thereproduction position is reproduced. The base view images include baseview anchor pictures 31, 32, 33, 34, and 35, the left view imagesinclude left view anchor pictures 131, 132, 133, 134, and 135, and theright view images include right view anchor pictures 231, 232, 233, 234,and 235.

Multi-view images may be reproduced and predicted (reconstructed)according to a GOP order. According to the reproduction order 30 of themulti-view video prediction structure, images included in a GOP 0 arereproduced according to views and then images included in a GOP 1 may bereproduced. That is, images included in each GOP may be reproduced inthe order of GOP 0, GOP 1, GOP 2, and GOP 3. According to a coding orderof the multi-view video prediction structure, the images included in theGOP 0 are predicted (reconstructed) according to views and then theimages included in the GOP 1 may be predicted (reconstructed). That is,the images included in each GOP may be reproduced in the order of GOP 0,GOP 1, GOP 2, and GOP 3.

According to the reproduction order 30 of the multi-view videoprediction structure, both inter-view prediction (inter-layerprediction) and inter prediction may be performed on images. In themulti-view video prediction structure, an image from which an arrowstarts is a reference image, and an image to which an arrow is directedis an image that is predicted by using the reference image.

A predicting result of the base view images may be encoded and then maybe output in the form of a base view image stream, and a predictionresult of the additional view images may be encoded and then may beoutput in the form of a layer bitstream. In addition, a predictionencoding result of the left view images may be output in a first layerbitstream and a prediction encoding result of the right view images maybe output in a second layer bitstream.

Only inter prediction is performed on base view images. That is, theanchor pictures 31, 32, 33, 34, and 35 that are I-picture type picturesdo not refer to different images, whereas the remaining images that areB-picture type images and b-picture type images are predicted withreference to different base view images. The B-picture type images arepredicted with reference to an I-picture type anchor picture having apreceding POC order and an I-picture type anchor picture having a laterPOC order. b-picture type images are predicted with reference to anI-picture type anchor picture having a preceding POC order and aB-picture type image having a later POC order or a B-picture type imagehaving a preceding POC order and an I-picture type anchor picture havinga later POC order.

Inter-view prediction (inter-layer prediction) referring to differentview images and inter prediction referring to the same view images arerespectively performed on the left view images and the right viewimages.

Inter-view prediction (inter-layer prediction) may be performed on theleft view anchor pictures 131, 132, 133, 134, and 135, respectively,with reference to the base view anchor pictures 31, 32, 33, 34, and 35having the same POC order. Inter-view prediction may be performed on theright view anchor pictures 231, 232, 233, 234, and 235, respectively,with reference to the base view anchor pictures 31, 32, 33, 34, and 35or the left view anchor pictures 131, 132, 133, 134, and 135 having thesame POC order. Inter-view prediction (inter-layer prediction) referringto different view images having the same POC order may be performed onremaining images among the left view images and the right view images,other than the anchor pictures 131, 132, 133, 134, 135, 231, 232, 233,234, and 235.

The remaining images among the left view images and the right viewimages, other than the anchor pictures 131, 132, 133, 134, 135, 231,232, 233, 234, and 235, are predicted with reference to the same viewimages.

However, the left view images and the right view images may not bepredicted with reference to an anchor picture having a previousreproduction order among additional view images of the same view. Thatis, for inter prediction of a current left view image, the left viewimages except for a left view anchor picture having a reproduction orderprevious to that of the current left view image may be referred to.Likewise, for inter prediction of a current right view image, the rightview images except for a right view anchor picture having a reproductionorder previous to that of the current right view image may be referredto.

For inter prediction of the current left view image, prediction may beperformed by not referring to a left view image that belongs to a GOPprevious to a current GPO to which the current left view belongs but byreferring to a left view image that belongs to the current GOP and is tobe reconstructed before the current left view image. The right viewimage is the same as described above.

The inter-layer video decoding apparatus 20 according to an exemplaryembodiment may reconstruct base view images, left view images, and rightview images according to the reproduction order 30 of the multi-viewvideo prediction structure shown in FIG. 3.

The left view images may be reconstructed via inter-view disparitycompensation referring to the base view images and inter-image motioncompensation referring to the left view images. The right view imagesmay be reconstructed via inter-view disparity compensation referring tothe base view images and the left view images and inter-image motioncompensation referring to the right view images. Reference images needto be first reconstructed for disparity compensation and motioncompensation of the left view images and the right view images.

For inter-image motion compensation of the left view images, the leftview images may be reconstructed via inter-image motion compensationreferring to reconstructed left view reference images. For inter-imagemotion compensation of the right view images, the right view images maybe reconstructed via inter-image motion compensation referring toreconstructed right view reference images.

For inter-image motion compensation of the current left view image, aleft view image that belongs to a GOP previous to a current GPO to whichthe current left view belongs may not be referred to, but only a leftview image that belongs to the current GOP and is to be reconstructedbefore the current left view image, may be referred to. The right viewimage is the same as described above.

FIG. 4 is a flowchart of a method of determining whether to performluminance compensation, in an inter-layer video decoding method,according to an exemplary embodiment of the inventive concept.

In operation 41, the inter-layer video decoding apparatus 20 maydetermine whether residual prediction is applicable. For example, when‘rpEnableFlag’ obtained from a second layer bitstream indicates 1, theresidual prediction determiner 24 may determine that residual predictionis applicable, and may determine whether to perform residual predictionin operation 42. Also, when ‘rpEnableFlag’ obtained from the secondlayer bitstream indicates 0, the residual prediction determiner 24 maydetermine that residual prediction is not applicable, and may omit anoperation of determining whether to perform residual prediction.

In operation 42, whether to perform residual prediction is determined.For example, when a value of ‘iv_res_pred_weight_idx’ included inresidual prediction information obtained by the residual predictiondeterminer 24 from the second layer bitstream indicates 0, the residualprediction determiner 24 may determine that it is not possible toperform residual prediction. Also, when a value of‘iv_res_pred_weight_idx’ obtained by the residual prediction determiner24 from the second layer bitstream indicates a value other than ‘0,’ theresidual prediction determiner 24 may determine to perform residualprediction.

In operation 42, the residual prediction determiner 24 may determine aweight value. For example, when iv_res_pred_weight_idx’ obtained by theresidual prediction determiner 24 from the second layer bitstreamindicates ‘1,’ the residual prediction determiner 24 may determine 1 asa weight value. Also, when iv_res_pred_weight_idx’ obtained by theresidual prediction determiner 24 from the second layer bitstreamindicates ‘2,’ the residual prediction determiner 24 may determine ½ asa weight value.

In operation 43, the luminance compensation determiner 26 may determinewhether to perform luminance compensation. For example, when ‘ic_flag’obtained by the luminance compensation determiner 26 from the secondlayer bitstream indicates 1, the luminance compensation determiner 26may determine to perform luminance compensation. Also, when ‘ic_flag’obtained by the luminance compensation determiner 26 from the secondlayer bitstream indicates 0, the luminance compensation determiner 26may determine not to perform luminance compensation.

In operation 44, the second layer decoder 28 may perform luminancecompensation based on determination of the luminance compensationdeterminer 26.

In operation 45, the second layer decoder 28 may perform residualprediction by applying a weight value determined by the residualprediction determiner 24. For example, when a weight value determined bythe residual prediction determiner 24 is 1, the second layer decoder 28may use an obtained residual without any change to decode a second layerblock. When a weight value determined by the residual predictiondeterminer 24 is ½, the second layer decoder 28 may apply the weightvalue of ½ to the obtained residual to decode the second layer block.

According to an exemplary embodiment of the inventive concept, whenapplying a weight value to a residual, the second layer decoder 28 mayapply different weight values according to respective color elements.For example, when a weight value determined by the second layer decoder28 is ½, the second layer decoder 28 may apply the weight value of ½ toa luma residual, and may apply no weight value to a chroma residual.

While an example of an operation of the inter-layer video decodingapparatus 20 to determine whether to perform luminance compensation andwhether to perform residual prediction is described above with referenceto operations 41 through 46, the method described with reference to FIG.4 may also be performed by the video encoding apparatus 10.

FIG. 5 illustrates an example of syntax for determining whether toperform luminance compensation in an inter-layer video decodingapparatus, according to an exemplary embodiment of the inventiveconcept.

For a current block, a syntax cu_extension 50 may include a conditionalstatement 51 used to determine whether to perform residual prediction ona second layer block. In the conditional statement 51, if residualprediction is applicable (rpEnableFlag==1), the residual predictiondeterminer 24 may obtain residual prediction information including‘iv_res_pred_weight_idx’ from a second layer bitstream. When‘iv_res_pred_weight_idx’ obtained by the residual prediction determiner24 indicates a value other than ‘0,’ the residual prediction determiner24 may determine to perform residual prediction, and the second layerdecoder 28 may perform residual prediction.

The syntax 50 for a current block, cu_extension( ), may include aconditional statement 52 used to determine whether to perform luminancecompensation on the second layer block. When the residual predictiondeterminer 24 determines not to perform residual prediction(iv_res_pred_weight_idx==0), the luminance compensation determiner 26may obtain luminance compensation information (ic_flag) from a secondlayer bitstream. When luminance compensation information (ic_flag)obtained by the luminance compensation determiner 26 indicates ‘0,’ theluminance compensation determiner 26 may determine not to performluminance compensation on the second layer block. Also, when luminancecompensation information (ic_flag) obtained by the luminancecompensation determiner 26 indicates ‘1,’ the luminance compensationdeterminer 26 may determine to perform luminance compensation on thesecond layer block.

FIG. 6 is a flowchart of a method of determining whether to applyresidual prediction in an inter-layer video decoding method according toan exemplary embodiment of the inventive concept.

In operation 61, the residual prediction determiner 24 may determinewhether residual prediction is applicable. The residual predictiondeterminer 24 may obtain prediction mode information of a second layerblock to determine whether a prediction mode of the second layer blockis a Merge mode or a SKIP mode. Also, the residual prediction determiner24 may obtain partition type information of the second layer block todetermine whether a partition type of a current block is 2N×2N.

When the residual prediction determiner 24 determines that theprediction mode of the second layer block is a Merge mode or a SKIPmode, and a partition type of the second layer block is 2N×2N, theresidual prediction determiner 24 may determine that residual predictionis applicable to the second layer block, and may determine whether toperform residual prediction in operation 62.

When the residual prediction determiner 24 determines that theprediction mode of the second layer block is not a Merge mode or a SKIPmode, or a partition type of the second layer block is not 2N×2N, theresidual prediction determiner 24 may determine that residual predictionis not applicable to the second layer block, and may end the process.

In operation 62, the residual prediction determiner 24 may determinewhether to perform residual prediction. For example, the residualprediction determiner 24 may determine whether to perform residualprediction based on residual prediction information. Also, the residualprediction determiner 24 may determine a weight value to be applied to aresidual during residual prediction.

In operation 63, the second layer decoder 28 may perform residualprediction by applying a weight value determined by the residualprediction determiner 24. For example, when a weight value determined bythe residual prediction determiner 24 is 1, the second layer decoder 28may use an obtained residual without any change to decode the secondlayer block. When a weight value determined by the residual predictiondeterminer 24 is ½, the second layer decoder 28 may apply the weightvalue of ½ to the obtained residual to decode the second layer block.

According to an exemplary embodiment of the inventive concept, whenapplying a weight value to a residual, the second layer decoder 28 mayapply different weight values according to respective color elements.For example, when a weight value determined by the second layer decoder28 is ½, the second layer decoder 28 may apply the weight value of ½ toa luma residual, and may apply no weight value to a chroma residual.

While an example of an operation of the inter-layer video decodingapparatus 20 to determine whether to perform luminance compensation andwhether to perform residual prediction is described with reference tooperations 61 through 63, the method described with reference to FIG. 6may also be performed by the video encoding apparatus 10.

FIG. 7 illustrates an example of syntax for determining whether to applyresidual prediction in an inter-layer video decoding apparatus,according to an exemplary embodiment of the inventive concept.

For a current block, a syntax coding unit( ) may include conditionalstatements 71, 72 used to determine whether to apply residual predictionto the current block.

In the conditional statement 71, when a second layer block is in a skipmode (skip_flag[x0][y0]==1), the residual prediction determiner 24 maydetermine to apply residual prediction, and obtain‘iv_res_pred_weight_idx’ from a second layer bitstream to determinewhether to perform residual prediction.

In the conditional statement 72, when the second layer block is in amerge mode (merge_flag[x0][y0]), and a partition mode is 2N×2N(PartMode==PART_2N×2N), the residual prediction determiner 24 may obtain‘iv_res_pred_weight_idx’ from a second layer bitstream to determinewhether to perform residual prediction.

According to an exemplary embodiment of the inventive concept, when avalue of ‘iv_res_pred_weight_idx’ obtained by the residual predictiondeterminer 24 from a second layer bitstream indicates 0, the residualprediction determiner 24 may determine that residual prediction may notbe performed. Also, when a value of ‘iv_res_pred_weight_idx’ obtained bythe residual prediction determiner 24 from the second layer bitstreamindicates a value other than ‘0,’ the residual prediction determiner 24may determine to perform residual prediction.

According to an exemplary embodiment of the inventive concept, theresidual prediction determiner 24 may determine a weight value. Forexample, when a value of ‘iv_res_pred_weight_idx’ obtained by theresidual prediction determiner 24 from a second layer bitstreamindicates ‘1,’ the residual prediction determiner 24 may determine 1 asa weight value. Also, when a value of ‘iv_res_pred_weight_idx’ obtainedby the residual prediction determiner 24 from a second layer bitstreamindicates ‘2,’ the residual prediction determiner 24 may determine ½ asa weight value.

According to an exemplary embodiment of the inventive concept, when theresidual prediction determiner 24 determines to perform residualprediction, the second layer decoder 28 may apply a weight valuedetermined by the residual prediction determiner 24 to a residual duringresidual prediction. For example, when a weight value determined by theresidual prediction determiner 24 is 1, the second layer decoder 28 mayuse an obtained residual without any change to decode the second layerblock. When a weight value determined by the residual predictiondeterminer 24 is ½, the second layer decoder 28 may apply the weightvalue of ½ to the obtained residual to decode the second layer block.

According to an exemplary embodiment of the inventive concept, whenapplying a weight value to a residual, the second layer decoder 28 mayapply different weight values according to respective color elements.For example, when a weight value determined by the second layer decoder28 is ½, the second layer decoder 28 may apply a weight value of ½ to aluma residual, and may apply no weight value to a chroma residual.

FIG. 8 is a flowchart of a method of determining whether to performresidual prediction in an inter-layer video decoding method, accordingto an exemplary embodiment of the inventive concept.

In operation 81, the residual prediction determiner 24 may determinewhether residual prediction is applicable. The residual predictiondeterminer 24 may obtain prediction mode information of a second layerblock to determine whether a prediction mode of the second layer blockis a Merge mode or a SKIP mode. The residual prediction determiner 24may obtain partition type information of the second layer block todetermine whether a partition type of the second layer block is a 2N×2N.Also, the residual prediction determiner 24 may obtain ‘ic_flag’ from asecond layer bitstream to determine whether luminance compensation isperformed on the second layer block.

When the residual prediction determiner 24 determines that theprediction mode of the second layer block is a Merge mode or a SKIPmode, and the partition type of the second layer block is 2N×2N, andluminance compensation is not performed (ic_flag==0), the residualprediction determiner 24 may determine that residual prediction isapplicable, and may determine whether to perform residual prediction inoperation 82.

When the residual prediction determiner 24 determines that theprediction mode of the second layer block is not a Merge mode or a SKIPmode, or the partition type of the second layer block is not 2N×2N, orluminance compensation is performed (ic_flag==1), the residualprediction determiner 24 may determine that residual prediction is notapplicable, and end the process.

In operation 82, the residual prediction determiner 24 may determinewhether to perform residual prediction. For example, the residualprediction determiner 24 may determine whether to perform residualprediction based on residual prediction information. Also, whenperforming residual prediction, the residual prediction determiner 24may determine a weight value to be applied to a residual.

In operation 83, the second layer decoder 28 may perform residualprediction by applying a weight value determined by the residualprediction determiner 24. For example, when a weight value determined bythe residual prediction determiner 24 is 1, the second layer decoder 28may use an obtained residual without any change to decode the secondlayer block. When a weight value determined by the residual predictiondeterminer 24 is ½, the second layer decoder 28 may apply the weightvalue of ½ to the obtained residual to decode the second layer block.

While an example of an operation of the inter-layer video decodingapparatus 20 to determine whether to perform residual prediction isdescribed above with reference to operations 81 through 83, the methoddescribed with reference to FIG. 8 may also be performed by the videoencoding apparatus 10.

FIG. 9 illustrates an example of syntax for determining whether to applyresidual prediction in an inter-layer video decoding apparatus,according to an exemplary embodiment of the inventive concept.

A syntax for a current block, coding unit( ), may include conditionalstatements 91, 92 used to determine whether to apply residual predictionto the current block.

In the conditional statement 91, when a second layer block is in a skipmode (skip_flag[x0][y0]==1), and a value of ‘ic_flag’ indicates 0(ic_flag==0), the residual prediction determiner 24 may determine toapply residual prediction, and obtain ‘iv_res_pred_weight_idx’ from asecond layer bitstream to determine whether to perform residualprediction.

In the conditional statement 92, when a second layer block is in a mergemode (merge_flag[x0][y0]), and a partition mode is 2N×2N(PartMode==PART_2N×2N), and a value of ‘ic_flag’ is 0 (ic_flag==0), theresidual prediction determiner 24 may obtain ‘iv_res_pred_weight_idx’from a second layer bitstream to determine whether to perform residualprediction.

FIG. 10 illustrates an example of syntax for applying a weight value toa residual in an inter-layer video decoding apparatus, according to anexemplary embodiment of the inventive concept.

According to an exemplary embodiment of the inventive concept, theresidual prediction determiner 24 may determine to perform residualprediction, and may determine a weight value by using weight informationobtained from residual prediction data. The second layer decoder 28 mayapply the weight value determined by the residual prediction determiner24 to a residual during residual prediction.

According to an exemplary embodiment of the inventive concept, whenapplying a weight value to a residual, the second layer decoder 28 mayapply different weight values according to respective color elements.For example, the second layer decoder 28 may apply a weight value to aluma residual, and may apply no weight value to a chroma residual.

102 of FIG. 10 illustrates an example of applying a weight value to aluma residual. The second layer decoder 28 may apply a weight value(ShiftVal) obtained by using weight information (iv_res_pred_weight_idx)to a luma residual so as to obtain a resultant value (predSamplesLX_L[x][y]) of residual prediction with respect to a luma component.

104 and 106 of FIG. 10 illustrate an example of not applying a weightvalue to a chroma residual. The second layer decoder 28 may not apply aweight value (ShiftVal) obtained by using weight information(iv_res_pred_weight_idx) to a chroma residual, and may obtain aresultant value (predSamplesLX_b [x][y], predSamplesLX_r [x][y]) ofresidual prediction with respect to a chroma component.

In the inter-layer video encoding apparatus 10 according to an exemplaryembodiment and the inter-layer video decoding apparatus 20 according toan exemplary embodiment, as described above, video data may be splitinto coding units having a tree structure, and coding units, predictionunits, and transformation units are used for inter-layer prediction orinter prediction on the coding units. Hereinafter, a video encodingmethod and apparatus and a video decoding method and apparatus based oncoding units having a tree structure according to an exemplaryembodiment will be described with reference to FIGS. 11 through 23.

In principle, during encoding/decoding for multi-layer video,encoding/decoding processes for first layer images and encoding/decodingprocesses for second layer images are separately performed. That is,when inter-layer prediction is performed on a multi-layer video,encoding/decoding results of a single-layer video may be referred toeach other, but separate encoding/decoding processes are performed forrespective single-layer videos.

For convenience of description, since a video encoding process and avideo decoding process based on a coding unit having a tree structure,which will be described with reference to FIGS. 11 through 23, areperformed on a single-layer video, only inter prediction and motioncompensation will be described. However, as described with reference toFIGS. 1A through 10, inter-layer prediction and compensation betweenbase layer images and second layer images are performed to encode/decodea video stream.

Thus, when the encoder 12 of the inter-layer video encoding apparatus 10according to an exemplary embodiment encodes a multi-layer video basedon a coding unit having a tree structure, in order to perform videoencoding on each respective single-layer video, the inter-layer videoencoding apparatus 10 includes as many video encoding apparatuses 100 ofFIG. 11 as the number of layers of the multi-layer video in order toencode a video such that each video encoding apparatus 100 may becontrolled to encode an assigned single-layer video. In addition, theinter-layer video encoding apparatus 10 may perform inter-viewprediction by using the encoding results of separate single-views ofeach video encoding apparatus 100. Thus, the encoder 12 of theinter-layer video encoding apparatus 10 may generate a base view videostream and a second layer video stream, in which the encoding resultsfor respective layers are recorded.

Similarly, when the decoder 26 of the inter-layer video decodingapparatus 20 according to an exemplary embodiment decodes a multi-layervideo based on a coding unit having a tree structure, in order toperform video decoding on the received first layer video stream andsecond layer video stream for each respective layer, the inter-layervideo decoding apparatus 20 may include as many video decodingapparatuses 200 of FIG. 13 as the number of layers of the multi-layervideo and the video decoding apparatuses 200 may be controlled toperform decoding on single-layer videos that are respectively assignedto the video decoding apparatuses 200. In addition, the inter-layervideo decoding apparatus 20 may perform inter-layer prediction by usingthe decoding result of separate single-layer of each video decodingapparatus 200. Thus, the decoder 26 of the inter-layer video decodingapparatus 20 may generate first layer images and second layer images,which are reconstructed for respective layers.

FIG. 11 is a block diagram of the video encoding apparatus 100 based oncoding units having a tree structure, according to an exemplaryembodiment of the inventive concept.

The video encoding apparatus 100 involving video prediction based oncoding units having a tree structure according to an exemplaryembodiment includes a largest coding unit (LCU) splitter 110, a codingunit determiner 120 and an outputter 130. For convenience ofdescription, the video encoding apparatus 100 involving video predictionbased on coding units according to a tree structure according to anexemplary embodiment will be abbreviated as ‘video encoding apparatus100’ below.

The LCU splitter 110 may divide a current picture based on a largestcoding unit (LCU) that is a coding unit having a maximum size for acurrent picture of an image. If the current picture is larger than theLCU, image data of the current picture may be split into the at leastone LCU. The LCU according to an exemplary embodiment may be a data unithaving a size of 32×32, 64×64, 128×128, 256×256, etc., wherein a shapeof the data unit is a square having a width and length in squares of 2.

A coding unit according to an exemplary embodiment may be characterizedby a maximum size and a depth. The depth denotes the number of times thecoding unit is spatially split from the LCU, and as the depth increases,deeper coding units according to depths may be split from the LCU to asmallest coding unit (SCU). A depth of the LCU is an uppermost depth anda depth of the SCU is a lowermost depth. Since a size of a coding unitcorresponding to each depth decreases as the depth of the LCU increases,a coding unit corresponding to an upper depth may include a plurality ofcoding units corresponding to lower depths.

As described above, the image data of the current picture is split intothe LCUs according to a maximum size of the coding unit, and each of theLCUs may include deeper coding units that are split according to depths.Since the LCU according to an exemplary embodiment is split according todepths, the image data of the space domain included in the LCU may behierarchically classified according to depths.

A maximum depth and a maximum size of a coding unit, which limit thetotal number of times a height and a width of the LCU are hierarchicallysplit, may be predetermined.

The coding unit determiner 120 encodes at least one split regionobtained by splitting a region of the LCU according to depths, anddetermines a depth to output a finally encoded image data according tothe at least one split region. In other words, the coding unitdeterminer 120 determines a depth by encoding the image data in thedeeper coding units according to depths, according to the LCU of thecurrent picture, and selecting a depth having the least encoding error.The determined depth and the encoded image data according to thedetermined depth are output to the outputter 130.

The image data in the LCU is encoded based on the deeper coding unitscorresponding to at least one depth equal to or below the maximum depth,and results of encoding the image data are compared based on each of thedeeper coding units. A depth having the least encoding error may beselected after comparing encoding errors of the deeper coding units. Atleast one depth may be selected for each LCU.

The size of the LCU is split as a coding unit is hierarchically splitaccording to depths, and the number of coding units increases. Also,even if coding units correspond to the same depth in one LCU, it isdetermined whether to split each of the coding units corresponding tothe same depth to a lower depth by measuring an encoding error of theimage data of the each coding unit, separately. Accordingly, even whenimage data is included in one LCU, the encoding errors may differaccording to regions in the one LCU, and thus the depths may differaccording to regions in the image data. Thus, one or more depths may bedetermined in one LCU, and the image data of the LCU may be dividedaccording to coding units of at least one depth.

Accordingly, the coding unit determiner 120 according to an exemplaryembodiment may determine coding units having a tree structure includedin the LCU. The ‘coding units having a tree structure’ according to anexemplary embodiment include coding units corresponding to a depthdetermined to be the depth, from among all deeper coding units includedin the LCU. A coding unit of a depth may be hierarchically determinedaccording to depths in the same region of the LCU, and may beindependently determined in different regions. Similarly, a depth in acurrent region may be independently determined from a depth in anotherregion.

A maximum depth according to an exemplary embodiment is an index relatedto the number of splitting times from a LCU to an SCU. A first maximumdepth according to an exemplary embodiment may denote the total numberof splitting times from the LCU to the SCU. A second maximum depthaccording to an exemplary embodiment may denote the total number ofdepth levels from the LCU to the SCU. For example, when a depth of theLCU is 0, a depth of a coding unit, in which the LCU is split once, maybe set to 1, and a depth of a coding unit, in which the LCU is splittwice, may be set to 2. Here, if the SCU is a coding unit in which theLCU is split four times, 5 depth levels of depths 0, 1, 2, 3, and 4exist, and thus the first maximum depth may be set to 4, and the secondmaximum depth may be set to 5.

Prediction encoding and transformation may be performed according to theLCU. The prediction encoding and the transformation are also performedbased on the deeper coding units according to a depth equal to or depthsless than the maximum depth, according to the LCU.

Since the number of deeper coding units increases whenever the LCU issplit according to depths, encoding, including the prediction encodingand the transformation, is performed on all of the deeper coding unitsgenerated as the depth increases. For convenience of description, theprediction encoding and the transformation will now be described basedon a coding unit of a current depth, in at least one LCU.

The video encoding apparatus 100 according to an exemplary embodimentmay variously select a size or shape of a data unit for encoding theimage data. In order to encode the image data, operations, such asprediction encoding, transformation, and entropy encoding, areperformed, and at this time, the same data unit may be used for alloperations or different data units may be used for each operation.

For example, the video encoding apparatus 100 may select not only acoding unit for encoding the image data, but also a data unit differentfrom the coding unit so as to perform the prediction encoding on theimage data in the coding unit.

In order to perform prediction encoding in the LCU, the predictionencoding may be performed based on a coding unit corresponding to adepth, i.e., based on a coding unit that is no longer split to codingunits corresponding to a lower depth. Hereinafter, the coding unit thatis no longer split and becomes a basis unit for prediction encoding willnow be referred to as a ‘prediction unit’. A partition obtained bysplitting the prediction unit may include a prediction unit or a dataunit obtained by splitting at least one of a height and a width of theprediction unit. A partition is a data unit where a prediction unit of acoding unit is split, and a prediction unit may be a partition havingthe same size as a coding unit.

For example, when a coding unit of 2N×2N (where N is a positive integer)is no longer split and becomes a prediction unit of 2N×2N, and a size ofa partition may be 2N×2N, 2N×N, N×2N, or N×N. Examples of a partitionmode include symmetrical partitions that are obtained by symmetricallysplitting a height or width of the prediction unit, partitions obtainedby asymmetrically splitting the height or width of the prediction unit,such as 1:n or n:1, partitions that are obtained by geometricallysplitting the prediction unit, and partitions having arbitrary shapes.

A prediction mode of the prediction unit may be at least one of an intramode, an inter mode, and a skip mode. For example, the intra mode or theinter mode may be performed on the partition of 2N×2N, 2N×N, N×2N, orN×N. Also, the skip mode may be performed only on the partition of2N×2N. The encoding is independently performed on one prediction unit ina coding unit, thereby selecting a prediction mode having a leastencoding error.

The video encoding apparatus 100 according to an exemplary embodimentmay also perform the transformation on the image data in a coding unitbased not only on the coding unit for encoding the image data, but alsobased on a data unit that is different from the coding unit. In order toperform the transformation in the coding unit, the transformation may beperformed based on a transformation unit having a size smaller than orequal to the coding unit. For example, the transformation unit mayinclude a data unit for an intra mode and a data unit for an inter mode.

The transformation unit in the coding unit may be recursively split intosmaller sized regions in the similar manner as the coding unit accordingto the tree structure. Thus, residual data in the coding unit may bedivided according to the transformation unit having the tree structureaccording to transformation depths.

A transformation depth indicating the number of splitting times to reachthe transformation unit by splitting the height and width of the codingunit may also be set in the transformation unit according to anexemplary embodiment. For example, in a current coding unit of 2N×2N, atransformation depth may be 0 when the size of a transformation unit is2N×2N, may be 1 when the size of the transformation unit is N×N, and maybe 2 when the size of the transformation unit is N/2×N/2. In otherwords, the transformation unit having the tree structure may be setaccording to the transformation depths.

Split information according to depths requires not only informationabout the depth, but also prediction-related information andtransformation-related information. Accordingly, the coding unitdeterminer 120 not only determines a depth having a least encodingerror, but also determines a partition mode in a prediction unit splitinto partitions, a prediction mode according to prediction units, and asize of a transformation unit for transformation.

Coding units having a tree structure in a LCU and methods of determininga prediction unit/partition, and a transformation unit, according to anexemplary embodiment, will be described in detail below with referenceto FIGS. 9 through 19.

The coding unit determiner 120 may measure an encoding error of deepercoding units according to depths by using Rate-Distortion Optimizationbased on Lagrangian multipliers.

The outputter 130 outputs the image data of the LCU, which is encodedbased on the at least one depth determined by the coding unit determiner120, and information about the encoding mode according to the depth, inbitstreams.

The encoded image data may be obtained by encoding residual data of animage.

The split information according to depths may include information aboutthe depth, about the partition mode in the prediction unit, theprediction mode, and the size of the transformation unit.

The information about the depth may be defined by using splittinginformation according to depths, which indicates whether encoding isperformed on coding units of a lower depth instead of a current depth.If the current depth of the current coding unit is the depth, thecurrent coding unit is encoded in coding units of the current depth, andthus the splitting information of the current depth may be defined notto split the current coding unit to a lower depth. Alternatively, if thecurrent depth of the current coding unit is not the depth, the encodingis performed on the coding unit of the lower depth, and thus thesplitting information of the current depth may be defined to split thecurrent coding unit to obtain the coding units of the lower depth.

If the current depth is not the depth, encoding is performed on thecoding unit that is split into the coding unit of the lower depth. Sinceat least one coding unit of the lower depth exists in one coding unit ofthe current depth, the encoding is repeatedly performed on each codingunit of the lower depth, and thus the encoding may be recursivelyperformed for the coding units having the same depth.

Since the coding units having a tree structure are determined for oneLCU, and information about at least one encoding mode is determined fora coding unit of a depth, at least one piece of splitting informationmay be determined for one LCU. Also, a depth of the data of the LCU maybe different according to locations since the image data ishierarchically split according to depths, and thus depth and splittinginformation may be set for the data.

Accordingly, the outputter 130 according to an exemplary embodiment mayassign encoding information about corresponding depth and encoding modeto at least one of the coding unit, the prediction unit, and a minimumunit included in the LCU.

The minimum unit according to an exemplary embodiment is a square dataunit obtained by splitting the SCU constituting the lowermost depth by4. Alternatively, the minimum unit according to an exemplary embodimentmay be a maximum square data unit that may be included in all of thecoding units, prediction units, partition units, and transformationunits included in the LCU.

For example, the encoding information output by the outputter 130 may beclassified into encoding information according to deeper coding units,and encoding information according to prediction units. The encodinginformation according to the deeper coding units may include theinformation about the prediction mode and about the size of thepartitions. The encoding information according to the prediction unitsmay include information about an estimated direction of an inter mode,about a reference image index of the inter mode, about a motion vector,about a chroma component of an intra mode, and about an interpolationmethod of the intra mode.

Information about a maximum size of the coding unit defined according topictures, slices, or GOPs, and information about a maximum depth may beinserted into a header of a bitstream, a sequence parameter set, or apicture parameter set.

Information about a maximum size of the transformation unit permittedwith respect to a current video, and information about a minimum size ofthe transformation unit may also be output through a header of abitstream, a sequence parameter set, or a picture parameter set. Theoutputter 130 may encode and output reference information related toprediction, prediction information, slice type information or the like.

According to a simplest exemplary embodiment of the video encodingapparatus 100, the deeper coding unit may be a coding unit obtained bydividing a height or width of a coding unit of an upper depth, which isone layer above, by two. In other words, when the size of the codingunit of the current depth is 2N×2N, the size of the coding unit of thelower depth is N×N. Also, the current coding unit having a size of 2N×2Nmay include a maximum of 4 of the coding units of the lower depth and ofa size of N×N.

Accordingly, the video encoding apparatus 100 may form the coding unitshaving the tree structure by determining coding units having an optimumshape and an optimum size for each LCU, based on the size of the LCU andthe maximum depth determined considering characteristics of the currentpicture. Also, since encoding may be performed on each LCU by using anyone of various prediction modes and transformations, an optimum encodingmode may be determined considering image characteristics of the codingunit of various image sizes.

Thus, if an image having a high resolution or a large data amount isencoded in a conventional macroblock, the number of macroblocks perpicture excessively increases. Accordingly, the number of pieces ofcompressed information generated for each macroblock increases, and thusit is difficult to transmit the compressed information and datacompression efficiency decreases. However, by using the video encodingapparatus, image compression efficiency may be increased since a codingunit is adjusted while considering characteristics of an image whileincreasing a maximum size of a coding unit while considering a size ofthe image.

The inter-layer video encoding apparatus 10 described with reference toFIG. 1A may include as many video encoding apparatuses 100 as the numberof layers in order to encode single-layer images for respective layersof a multi-layer video. For example, the first layer encoder 12 mayinclude a single video encoding apparatus 100 and the second layerencoder 14 may include as many video encoding apparatuses 100 as thenumber of second layers.

When the video encoding apparatus 100 encodes first layer images, thecoding unit determiner 120 may determine a prediction unit for interprediction for each respective coding unit having a tree structure foreach largest coding unit and may perform inter prediction for eachrespective prediction unit.

When the video encoding apparatus 100 encodes second layer images, thecoding unit determiner 120 may also determine a prediction unit and acoding unit having a tree structure for each largest coding unit and mayperform inter prediction for each respective prediction unit.

The video encoding apparatus 100 may encode a brightness differencebetween first and second layer images for compensating for thebrightness difference. However, whether to perform brightnesscompensation may be determined according to an encoding mode of a codingunit. For example, the brightness compensation may be performed only ona prediction unit of 2N×2N.

FIG. 12 is a block diagram of the video decoding apparatus 200 based oncoding units having a tree structure, according to various exemplaryembodiments.

The video decoding apparatus 200 that involves video prediction based oncoding units having a tree structure according to an exemplaryembodiment includes a receiver 210, an image data and encodinginformation extractor 220, and an image data decoder 230. Forconvenience of description, the video decoding apparatus 200 involvingvideo prediction based on coding units according to a tree structureaccording to an exemplary embodiment will be abbreviated as ‘videodecoding apparatus 200’ below.

Definitions of various terms, such as a coding unit, a depth, aprediction unit, a transformation unit, and various splittinginformation, for decoding operations of the video decoding apparatus 200according to an exemplary embodiment are identical to those describedwith reference to FIG. 11 and the video encoding apparatus 100.

The receiver 210 receives and parses a bitstream of an encoded video.The image data and encoding information extractor 220 extracts encodedimage data for each coding unit from the parsed bitstream, wherein thecoding units have a tree structure according to each LCU, and outputsthe extracted image data to the image data decoder 230. The image dataand encoding information extractor 220 may extract information about amaximum size of a coding unit of a current picture, from a header aboutthe current picture, a sequence parameter set, or a picture parameterset.

The image data and encoding information extractor 220 extracts finaldepth and splitting information for the coding units having a treestructure according to each LCU, from the parsed bitstream. Theextracted final depth and splitting information are output to the imagedata decoder 230. In other words, the image data in a bit stream issplit into the LCU so that the image data decoder 230 decodes the imagedata for each LCU.

The depth and splitting information according to the LCU may be set forat least one piece of information corresponding to the depth, andsplitting information according to the depth may include informationabout a partition mode of a corresponding coding unit corresponding tothe depth, information about a prediction mode, and splittinginformation of a transformation unit. Also, splitting informationaccording to depths may be extracted as the depth information.

The depth and splitting information according to each LCU extracted bythe image data and encoding information extractor 220 is a depth andsplitting information determined to generate a minimum encoding errorwhen an encoder, such as the video encoding apparatus 100 according toan exemplary embodiment, repeatedly performs encoding for each deepercoding unit according to depths according to each LCU. Accordingly, thevideo decoding apparatus 200 may reconstruct an image by decoding thedata according to an encoding mode that generates the minimum encodingerror.

Since the depth and the encoding information about an encoding mode maybe assigned to a predetermined data unit from among a correspondingcoding unit, a prediction unit, and a minimum unit, the image data andencoding information extractor 220 may extract the depth and thesplitting information according to the predetermined data units. If adepth and splitting information of a corresponding LCU are recordedaccording to predetermined data units, the predetermined data units towhich the same depth and the same splitting information are assigned maybe inferred to be the data units included in the same LCU.

The image data decoder 230 reconstructs the current picture by decodingthe image data in each LCU based on the depth and the splittinginformation according to the LCUs. In other words, the image datadecoder 230 may decode the encoded image data based on the extractedinformation about the partition mode, the prediction mode, and thetransformation unit for each coding unit from among the coding unitshaving the tree structure included in each LCU. A decoding process mayinclude a prediction process including intra prediction and motioncompensation, and an inverse transformation process.

The image data decoder 230 may perform intra prediction or motioncompensation according to a partition and a prediction mode of eachcoding unit, based on the information about the partition mode and theprediction mode of the prediction unit of the coding unit according todepths.

In addition, the image data decoder 230 may read information about atransformation unit having a tree structure for each coding unit so asto perform inverse transformation based on transformation units for eachcoding unit, for inverse transformation for each LCU. Via the inversetransformation, a pixel value of the space domain of the coding unit maybe reconstructed.

The image data decoder 230 may determine a depth of a current LCU byusing splitting information according to depths. If the splittinginformation indicates that image data is no longer split in the currentdepth, the current depth is the depth. Accordingly, the image datadecoder 230 may decode encoded image data in the current LCU by usingthe information about the partition mode of the prediction unit, theinformation about the prediction mode, and the splitting information ofthe transformation unit for each coding unit corresponding to the depth.

In other words, data units containing the encoding information includingthe same splitting information may be gathered by observing the encodinginformation set for the predetermined data unit from among the codingunit, the prediction unit, and the minimum unit, and the gathered dataunits may be considered to be one data unit to be decoded by the imagedata decoder 230 in the same encoding mode. As such, the current codingunit may be decoded by obtaining the information about the encoding modefor each coding unit.

The inter-layer video decoding apparatus 20 described with reference toFIG. 2A may include as many video decoding apparatuses 200 as the numberof views in order to decode the received first layer image stream andsecond layer image stream to reconstruct first layer images and secondlayer images.

When a first layer image stream is received, the image data decoder 230of the video decoding apparatus 200 may split samples of first layerimages that are extracted from the first layer image stream by theextractor 220 into coding units having a tree structure of a largestcoding unit. The image data decoder 230 may perform motion compensationon respective prediction units for inter prediction for each respectivecoding unit having a tree structure of the samples of the first layerimages, to reconstruct the first layer images.

When a second layer image stream is received, the image data decoder 230of the video decoding apparatus 200 may split samples of second layerimages that are extracted from the second layer image stream by theextractor 220 into coding units having a tree structure of a largestcoding unit. The image data decoder 230 may perform motion compensationon respective prediction units for inter prediction of the samples ofthe second layer images to reconstruct the second layer images.

The extractor 220 may obtain information relating to a brightness errorfrom a bitstream in order to compensate for a brightness differencebetween first and second layer images. However, whether to performbrightness compensation may be determined according to an encoding modeof a coding unit. For example, the brightness compensation may beperformed only on a prediction unit of 2N×2N.

The video decoding apparatus 200 may obtain information about a codingunit that generates the minimum encoding error when encoding isrecursively performed for each largest coding unit, and may use theinformation to decode the current picture. In other words, the codingunits having the tree structure determined to be the optimum codingunits in each largest coding unit may be decoded.

Accordingly, even if image data has high resolution and a large amountof data, the image data may be efficiently decoded and reconstructed byusing a size of a coding unit and an encoding mode, which are adaptivelydetermined according to characteristics of the image, by usinginformation about optimum splitting received from an encoder.

FIG. 13 is a diagram for describing a concept of coding units accordingto various exemplary embodiments.

A size of a coding unit may be expressed by width×height, and may be64×64, 32×32, 16×16, and 8×8. A coding unit of 64×64 may be split intopartitions of 64×64, 64×32, 32×64, or 32×32, and a coding unit of 32×32may be split into partitions of 32×32, 32×16, 16×32, or 16×16, a codingunit of 16×16 may be split into partitions of 16×16, 16×8, 8×16, or 8×8,and a coding unit of 8×8 may be split into partitions of 8×8, 8×4, 4×8,or 4×4.

In video data 310, a resolution is 1920×1080, a maximum size of a codingunit is 64, and a maximum depth is 2. In video data 320, a resolution is1920×1080, a maximum size of a coding unit is 64, and a maximum depth is3. In video data 330, a resolution is 352×288, a maximum size of acoding unit is 16, and a maximum depth is 1. The maximum depth shown inFIG. 13 denotes a total number of splits from a LCU to a minimum codingunit.

If a resolution is high or a data amount is large, a maximum size of acoding unit may be large so as to not only increase encoding efficiencybut also to accurately reflect characteristics of an image. Accordingly,the maximum size of the coding unit of the video data 310 and 320 havinga higher resolution than the video data 330 may be 64.

Since the maximum depth of the video data 310 is 2, coding units 315 ofthe vide data 310 may include from a LCU having a long axis size of 64,to coding units having long axis sizes of 32 and 16 since depths areincreased to two layers by splitting the LCU twice. Since the maximumdepth of the video data 330 is 1, coding units 335 of the video data 330may include a LCU having a long axis size of 16, and coding units havinga long axis size of 8 since depths are increased to one layer bysplitting the LCU once.

Since the maximum depth of the video data 320 is 3, coding units 325 ofthe video data 320 may include from a LCU having a long axis size of 64,and to coding units having long axis sizes of 32, 16, and 8 since thedepths are increased to 3 layers by splitting the LCU three times. As adepth increases, detailed information may be precisely expressed.

FIG. 14 is a block diagram of an image encoder 400 based on codingunits, according to various exemplary embodiments.

The image encoder 400 according to an exemplary embodiment performsoperations necessary for encoding image data in the coding unitdeterminer 120 of the video encoding apparatus 100. In other words, anintra predictor 420 performs intra prediction on coding units in anintra mode according to prediction units, from among a current image405, and an inter predictor 415 performs inter prediction on codingunits in an inter mode by using the current image 405 and a referenceimage obtained from a reconstructed picture buffer 410 according toprediction units. The current image 405 may be split into LCUs and thenthe LCUs may be sequentially encoded. In this regard, the LCUs that areto be split into coding units having a tree structure may be encoded.

Residue data is generated by removing prediction data regarding codingunits of each mode that is output from the intra predictor 420 or theinter predictor 415 from data regarding encoded coding units of thecurrent image 405, and is output as a quantized transformationcoefficient according to transformation units through a transformer 425and a quantizer 430. The quantized transformation coefficient isreconstructed as the residue data in a space domain through adequantizer 445 and an inverse transformer 450. The reconstructedresidue data in the space domain is added to prediction data for codingunits of each mode that is output from the intra predictor 420 or theinter predictor 415 and thus is reconstructed as data in a space domainfor coding units of the current image 405. The reconstructed data in thespace domain is generated as reconstructed images through a de-blocker455 and an SAO performer 460 and the reconstructed images are stored inthe reconstructed picture buffer 410. The reconstructed images stored inthe reconstructed picture buffer 410 may be used as reference images forinter prediction of another image. The transformation coefficientquantized by the transformer 425 and the quantizer 430 may be output asa bitstream 440 through an entropy encoder 435.

In order for the image encoder 400 according to an exemplary embodimentto be applied in the video encoding apparatus 100, all elements of theimage encoder 400, i.e., the inter predictor 415, the intra predictor420, the transformer 425, the quantizer 430, the entropy encoder 435,the dequantizer 445, the inverse transformer 450, the de-blocker 455,and the SAO performer 460, perform operations based on each coding unitamong coding units having a tree structure according to each LCU.

Specifically, the intra predictor 420 and the inter predictor 415 maydetermine a partition mode and a prediction mode of each coding unitamong the coding units having a tree structure in consideration of amaximum size and a maximum depth of a current LCU, and the transformer425 may determine whether to split a transformation unit having a quadtree structure in each coding unit among the coding units having a treestructure.

FIG. 15 is a block diagram of an image decoder 500 based on codingunits, according to various exemplary embodiments.

An entropy decoder 515 parses encoded image data to be decoded andinformation about encoding required for decoding from a bitstream 505.The encoded image data is a quantized transformation coefficient fromwhich residue data is reconstructed by a dequantizer 520 and an inversetransformer 525.

An intra predictor 540 performs intra prediction on coding units in anintra mode according to each prediction unit. An inter predictor 535performs inter prediction on coding units in an inter mode from amongthe current image for each prediction unit by using a reference imageobtained from a reconstructed picture buffer 530.

Prediction data and residue data regarding coding units of each mode,which passed through the intra predictor 540 and the inter predictor535, are summed, and thus data in a space domain regarding coding unitsof the current image 405 may be reconstructed, and the reconstructeddata in the space domain may be output as a reconstructed image 560through a de-blocker 545 and an SAO performer 550. Reconstructed imagesstored in the reconstructed picture buffer 530 may be output asreference images.

In order to decode the image data in the image data decoder 230 of thevideo decoding apparatus 200, operations after the entropy decoder 515of the image decoder 500 according to an exemplary embodiment may beperformed.

In order for the image decoder 500 to be applied in the video decodingapparatus 200 according to an exemplary embodiment, all elements of theimage decoder 500, i.e., the entropy decoder 515, the dequantizer 520,the inverse transformer 525, the intra predictor 540, the interpredictor 535, the de-blocker 545, and the SAO performer 550 may performoperations based on coding units having a tree structure for each LCU.

In particular, the intra predictor 540 and the inter predictor 535 maydetermine a partition mode and a prediction mode for each of the codingunits having a tree structure, and the inverse transformer 525 maydetermine whether to split a transformation unit having a quad treestructure for each of the coding units.

The encoding operation of FIG. 14 and the decoding operation of FIG. 15describe video stream encoding and decoding operations in a singlelayer, respectively. Thus, if the encoder 12 of FIG. 1A encodes videostreams of two or more layers, the image encoder 400 may be provided foreach layer. Similarly, if the decoder 26 of FIG. 2A decodes videostreams of two or more layers, the image decoder 500 may be provided foreach layer.

FIG. 16 is a diagram illustrating deeper coding units according todepths, and partitions, according to various exemplary embodiments.

The video encoding apparatus 100 according to an exemplary embodimentand the video decoding apparatus 200 according to an exemplaryembodiment use hierarchical coding units so as to considercharacteristics of an image. A maximum height, a maximum width, and amaximum depth of coding units may be adaptively determined according tothe characteristics of the image, or may be differently set by a user.Sizes of deeper coding units according to depths may be determinedaccording to the predetermined maximum size of the coding unit.

In a hierarchical structure 600 of coding units, according to anexemplary embodiment, the maximum height and the maximum width of thecoding units are each 64, and the maximum depth is 3. In this case, themaximum depth refers to a total number of times the coding unit is splitfrom the LCU to the SCU. Since a depth increases along a vertical axisof the hierarchical structure 600, a height and a width of the deepercoding unit are each split. Also, a prediction unit and partitions,which are bases for prediction encoding of each deeper coding unit, areshown along a horizontal axis of the hierarchical structure 600.

In other words, a coding unit 610 is a LCU in the hierarchical structure600, wherein a depth is 0 and a size, i.e., a height by width, is 64×64.The depth increases along the vertical axis, and a coding unit 620having a size of 32×32 and a depth of 1, a coding unit 630 having a sizeof 16×16 and a depth of 2, and a coding unit 640 having a size of 8×8and a depth of 3 exist. The coding unit 640 having a size of 8×8 and adepth of 3 is an SCU.

The prediction unit and the partitions of a coding unit are arrangedalong the horizontal axis according to each depth. In other words, ifthe coding unit 610 having a size of 64×64 and a depth of 0 is aprediction unit, the prediction unit may be split into partitionsincluded in the encoding unit 610, i.e. a partition 610 having a size of64×64, partitions 612 having the size of 64×32, partitions 614 havingthe size of 32×64, or partitions 616 having the size of 32×32.

Similarly, a prediction unit of the coding unit 620 having the size of32×32 and the depth of 1 may be split into partitions included in thecoding unit 620, i.e. a partition 620 having a size of 32×32, partitions622 having a size of 32×16, partitions 624 having a size of 16×32, andpartitions 626 having a size of 16×16.

Similarly, a prediction unit of the coding unit 630 having the size of16×16 and the depth of 2 may be split into partitions included in thecoding unit 630, i.e. a partition 630 having a size of 16×16, partitions632 having a size of 16×8, partitions 634 having a size of 8×16, andpartitions 636 having a size of 8×8.

Similarly, a prediction unit of the coding unit 640 having the size of8×8 and the depth of 3 may be split into partitions included in thecoding unit 640, i.e. a partition 640 having a size of 8×8, partitions642 having a size of 8×4, partitions 644 having a size of 4×8, andpartitions 646 having a size of 4×4.

In order to determine a depth of the coding units constituting the LCU610, the coding unit determiner 120 of the video encoding apparatus 100according to an exemplary embodiment performs encoding for coding unitscorresponding to each depth included in the LCU 610.

A number of deeper coding units according to depths including data inthe same range and the same size increases as the depth increases. Forexample, four coding units corresponding to a depth of 2 are required tocover data that is included in one coding unit corresponding to a depthof 1. Accordingly, in order to compare encoding results of the same dataaccording to depths, the coding unit corresponding to the depth of 1 andfour coding units corresponding to the depth of 2 are each encoded.

In order to perform encoding for a current depth from among the depths,a least encoding error may be selected for the current depth byperforming encoding for each prediction unit in the coding unitscorresponding to the current depth, along the horizontal axis of thehierarchical structure 600. Alternatively, the minimum encoding errormay be searched for by comparing the least encoding errors according todepths, by performing encoding for each depth as the depth increasesalong the vertical axis of the hierarchical structure 600. A depth and apartition having the minimum encoding error in the LCU 610 may beselected as the final depth and a partition mode of the LCU 610.

FIG. 17 is a diagram for describing a relationship between a coding unitand transformation units, according to various exemplary embodiments.

The video encoding apparatus 100 according to an exemplary embodiment orthe video decoding apparatus 200 according to an exemplary embodimentencodes or decodes an image according to coding units having sizessmaller than or equal to a LCU for each LCU. Sizes of transformationunits for transformation during encoding may be selected based on dataunits that are not larger than a corresponding coding unit.

For example, in the video encoding apparatus 100 or the video decodingapparatus 200, if a size of the coding unit 710 is 64×64, transformationmay be performed by using the transformation units 720 having a size of32×32.

Also, data of the coding unit 710 having the size of 64×64 may beencoded by performing the transformation on each of the transformationunits having the size of 32×32, 16×16, 8×8, and 4×4, which are smallerthan 64×64, and then a transformation unit having the least coding errormay be selected.

FIG. 18 is a diagram for describing encoding information of coding unitscorresponding to a depth, according to various exemplary embodiments.

The outputter 130 of the video encoding apparatus 100 according to anexemplary embodiment may encode and transmit information 800 about apartition mode, information 810 about a prediction mode, and information820 about a size of a transformation unit for each coding unitcorresponding to a depth, as splitting information.

The information 800 indicates information about a mode of a partitionobtained by splitting a prediction unit of a current coding unit,wherein the partition is a data unit for prediction encoding the currentcoding unit. For example, a current coding unit CU_0 having a size of2N×2N may be split into any one of a partition 802 having a size of2N×2N, a partition 804 having a size of 2N×N, a partition 806 having asize of N×2N, and a partition 808 having a size of N×N. Here, theinformation 800 about the partition mode is set to indicate one of thepartition 804 having a size of 2N×N, the partition 806 having a size ofN×2N, and the partition 808 having a size of N×N.

The information 810 indicates a prediction mode of each partition. Forexample, the information 810 may indicate a mode of prediction encodingperformed on a partition indicated by the information 800, i.e., anintra mode 812, an inter mode 814, or a skip mode 816.

The information 820 indicates a transformation unit to be based on whentransformation is performed on a current coding unit. For example, thetransformation unit may be one of a first intra transformation unit 822,a second intra transformation unit 824, a first inter transformationunit 826, and a second inter transformation unit 828.

The image data and encoding information extractor 220 of the videodecoding apparatus 200 according to an exemplary embodiment may extractand use the information 800, 810, and 820 for decoding, according toeach deeper coding unit.

FIG. 19 is a diagram of deeper coding units according to depths,according to various exemplary embodiments.

Splitting information may be used to indicate a change of a depth. Thesplitting information indicates whether a coding unit of a current depthis split into coding units of a lower depth.

A prediction unit 910 for prediction encoding a coding unit 900 having adepth of 0 and a size of 2N_0×2N_0 may include partitions of a partitionmode 912 having a size of 2N_0×2N_0, a partition mode 914 having a sizeof 2N_0×N_0, a partition mode 916 having a size of N_0×2N_0, and apartition mode 918 having a size of N_0×N_0. FIG. 19 only illustratesthe partition modes 912 through 918 which are obtained by symmetricallysplitting the prediction unit 910, but a partition mode is not limitedthereto, and the partitions of the prediction unit 910 may includeasymmetrical partitions, partitions having a predetermined shape, andpartitions having a geometrical shape.

Prediction encoding is repeatedly performed on one partition having asize of 2N_0×2N_0, two partitions having a size of 2N_0×N_0, twopartitions having a size of N_0×2N_0, and four partitions having a sizeof N_0×N_0, according to each partition mode. The prediction encoding inan intra mode and an inter mode may be performed on the partitionshaving the sizes of 2N_0×2N_0, N_0×2N_0, 2N_0×N_0, and N_0×N_0. Theprediction encoding in a skip mode is performed only on the partitionhaving the size of 2N_0×2N_0.

If an encoding error is smallest in one of the partition modes 912through 916, the prediction unit 910 may not be split into a lowerdepth.

If the encoding error is the smallest in the partition mode 918, a depthis changed from 0 to 1 to split the partition mode 918 in operation 920,and encoding is repeatedly performed on coding units 930 having a depthof 2 and a size of N_0×N_0 to search for a minimum encoding error.

A prediction unit 940 for prediction encoding the coding unit 930 havinga depth of 1 and a size of 2N_1×2N_1 (=N_0×N_0) may include partitionsof a partition mode 942 having a size of 2N_1×2N_1, a partition mode 944having a size of 2N_1×N_1, a partition mode 946 having a size ofN_1×2N_1, and a partition mode 948 having a size of N_1×N_1.

If an encoding error is the smallest in the partition mode 948, a depthis changed from 1 to 2 to split the partition mode 948 in operation 950,and encoding is repeatedly performed on coding units 960, which have adepth of 2 and a size of N_2×N_2 to search for a minimum encoding error.

When a maximum depth is d, split operation according to each depth maybe performed up to when a depth becomes d−1, and splitting informationmay be encoded as up to when a depth is one of 0 to d−2. In other words,when encoding is performed up to when the depth is d−1 after a codingunit corresponding to a depth of d−2 is split in operation 970, aprediction unit 990 for prediction encoding a coding unit 980 having adepth of d−1 and a size of 2N_(d−1)×2N_(d−1) may include partitions of apartition mode 992 having a size of 2N_(d−1)×2N_(d−1), a partition mode994 having a size of 2N (d−1)×N_(d−1), a partition mode 996 having asize of N_(d−1)×2N_(d−1), and a partition mode 998 having a size ofN_(d−1)×N_(d−1).

Prediction encoding may be repeatedly performed on one partition havinga size of 2N_(d−1)×2N_(d−1), two partitions having a size of2N_(d−1)×N_(d−1), two partitions having a size of N_(d−1)×2N_(d−1), fourpartitions having a size of N_(d−1)×N_(d−1) from among the partitionmodes to search for a partition mode having a minimum encoding error.

Even when the partition mode 998 having a size of N_(d−1)×N_(d−1) hasthe minimum encoding error, since a maximum depth is d, a coding unitCU_(d−1) having a depth of d−1 is no longer split to a lower depth, anda depth for the coding units constituting a current LCU 900 isdetermined to be d−1 and a partition mode of the current LCU 900 may bedetermined to be N_(d−1)×N_(d−1). Also, since the maximum depth is d,splitting information for the coding unit 952 having a depth of d−1 isnot set.

A data unit 999 may be a ‘minimum unit’ for the current LCU. A minimumunit according to an exemplary embodiment may be a square data unitobtained by splitting an SCU of a lowermost depth, by 4. By performingthe encoding repeatedly, the video encoding apparatus 100 according toan exemplary embodiment may select a depth having the least encodingerror by comparing encoding errors according to depths of the codingunit 900 to determine a depth, and set a corresponding partition modeand a prediction mode as an encoding mode of the depth.

As such, the minimum encoding errors according to depths are compared inall of the depths of 1 through d, and a depth having the least encodingerror may be determined as a depth. The depth, the partition mode of theprediction unit, and the prediction mode may be encoded and transmittedas information about an encoding mode. Also, since a coding unit issplit from a depth of 0 to a depth, only splitting information of thedepth is set to 0, and splitting information of depths excluding thedepth is set to 1.

The image data and encoding information extractor 220 of the videodecoding apparatus 200 according to an exemplary embodiment may extractand use the information about the depth and the prediction unit of thecoding unit 900 to decode the coding unit 912. The video decodingapparatus 200 according to an exemplary embodiment may determine adepth, in which splitting information is 0, as a depth by usingsplitting information according to depths, and use information about thecorresponding depth for decoding.

FIGS. 20 through 22 are diagrams for describing a relationship betweencoding units, prediction units, and transformation units, according tovarious exemplary embodiments.

The coding units 1010 are coding units corresponding to depthsdetermined by the video encoding apparatus 100 according to an exemplaryembodiment, in a LCU. The prediction units 1060 are partitions ofprediction units of each of coding units corresponding to depths, amongthe coding units 1010, and the transformation units 1070 aretransformation units of the coding units corresponding to depths.

When a depth of a LCU is 0 in the coding units 1010, depths of codingunits 1012 and 1054 are 1, depths of coding units 1014, 1016, 1018,1028, 1050, and 1052 are 2, depths of coding units 1020, 1022, 1024,1026, 1030, 1032, and 1048 are 3, and depths of coding units 1040, 1042,1044, and 1046 are 4.

In the prediction units 1060, some partitions 1014, 1016, 1022, 1032,1048, 1050, 1052, and 1054 are obtained by splitting the coding units.In other words, partition modes in the coding units 1014, 1022, 1050,and 1054 have a size of 2N×N, partition modes in the coding units 1016,1048, and 1052 have a size of N×2N, and a partition mode of the codingunit 1032 has a size of N×N. Prediction units and partitions of thecoding units 1010 are smaller than or equal to each coding unit.

Transformation or inverse transformation is performed on image data ofthe coding unit 1052 in the transformation units 1070 in a data unitthat is smaller than the coding unit 1052. Also, the coding units 1014,1016, 1022, 1032, 1048, 1050, and 1052 in the transformation units 1070are different from those in the prediction units 1060 in terms of sizesand shapes. In other words, the video encoding and decoding apparatuses100 and 200 according to an exemplary embodiment may perform intraprediction, motion estimation, motion compensation, transformation, andinverse transformation individually on a data unit in the same codingunit.

Accordingly, encoding is recursively performed on each of coding unitshaving a hierarchical structure in each region of a LCU to determine anoptimum coding unit, and thus coding units having a recursive treestructure may be obtained. Encoding information may include splittinginformation about a coding unit, information about a partition mode,information about a prediction mode, and information about a size of atransformation unit. Table 1 shows the encoding information that may beset by the video decoding apparatus 200.

TABLE 1 Split Information 0 Split (Encoding on Coding Unit having Sizeof 2N × 2N and Current Depth of d) Information 1 Prediction PartitionType Size of Transformation Unit Repeatedly Mode Encode Coding IntraSymmetrical Asymmetrical Split Split Units having Inter PartitionPartition Information 0 of Information 1 of Lower Depth Skip (Only TypeType Transformation Transformation of d + 1 2N × 2N) Unit Unit 2N × 2N2N × nU 2N × 2N N × N 2N × N 2N × nD (Symmetrical N × 2N nL × 2N Type) N× N nR × 2N N/2 × N/2 (Asymmetrical Type)

The outputter 130 of the video encoding apparatus 100 according to anexemplary embodiment may output the encoding information about thecoding units having a tree structure, and the encoding informationextractor 220 of the video decoding apparatus 200 according to anexemplary embodiment may extract the encoding information about thecoding units having a tree structure from a received bitstream.

Splitting information indicates whether a current coding unit is splitinto coding units of a lower depth. If splitting information of acurrent depth d is 0, a depth, in which a current coding unit is nolonger split into a lower depth, is a depth, and thus information abouta partition mode, prediction mode, and a size of a transformation unitmay be defined for the depth. If the current coding unit is furthersplit according to the splitting information, encoding is independentlyperformed on four split coding units of a lower depth.

A prediction mode may be one of an intra mode, an inter mode, and a skipmode. The intra mode and the inter mode may be defined in all partitionmodes, and the skip mode is defined only in a partition mode having asize of 2N×2N.

The information about the partition mode may indicate symmetricalpartition modes having sizes of 2N×2N, 2N×N, N×2N, and N×N, which areobtained by symmetrically splitting a height or a width of a predictionunit, and asymmetrical partition modes having sizes of 2N×nU, 2N×nD,nL×2N, and nR×2N, which are obtained by asymmetrically splitting theheight or width of the prediction unit. The asymmetrical partition modeshaving the sizes of 2N×nU and 2N×nD may be respectively obtained bysplitting the height of the prediction unit in 1:3 and 3:1, and theasymmetrical partition modes having the sizes of nL×2N and nR×2N may berespectively obtained by splitting the width of the prediction unit in1:3 and 3:1

The size of the transformation unit may be set to be two types in theintra mode and two types in the inter mode. In other words, if splittinginformation of the transformation unit is 0, the size of thetransformation unit may be 2N×2N, which is the size of the currentcoding unit. If splitting information of the transformation unit is 1,the transformation units may be obtained by splitting the current codingunit. Also, if a partition mode of the current coding unit having thesize of 2N×2N is a symmetrical partition mode, a size of atransformation unit may be N×N, and if the partition mode of the currentcoding unit is an asymmetrical partition mode, the size of thetransformation unit may be N/2×N/2.

The encoding information about coding units having a tree structure,according to an exemplary embodiment, may include at least one of acoding unit corresponding to a depth, a prediction unit, and a minimumunit. The coding unit corresponding to the depth may include at leastone of a prediction unit and a minimum unit containing the same encodinginformation.

Accordingly, it is determined whether adjacent data units are includedin the same coding unit corresponding to the depth by comparing encodinginformation of the adjacent data units. Also, a corresponding codingunit corresponding to a depth is determined by using encodinginformation of a data unit, and thus a distribution of depths in a LCUmay be determined.

Accordingly, if a current coding unit is predicted based on encodinginformation of adjacent data units, encoding information of data unitsin deeper coding units adjacent to the current coding unit may bedirectly referred to and used.

Alternatively, when a current coding unit is prediction encoded byreferring to adjacent coding units, data adjacent to the current codingunit in the coding units according to depths may be searched by usingencoding information of the adjacent coding units according to depths,and the searched data adjacent to the current coding unit may bereferred to for prediction encoding.

FIG. 23 is a diagram for describing a relationship between a codingunit, a prediction unit, and a transformation unit, according toencoding mode information of Table 1.

A LCU 1300 includes coding units 1302, 1304, 1306, 1312, 1314, 1316, and1318 of depths. Here, since the coding unit 1318 is a coding unit of adepth, splitting information may be set to 0. Information about apartition mode of the coding unit 1318 having a size of 2N×2N may be setto be one of a partition mode 1322 having a size of 2N×2N, a partitionmode 1324 having a size of 2N×N, a partition mode 1326 having a size ofN×2N, a partition mode 1328 having a size of N×N, a partition mode 1332having a size of 2N×nU, a partition mode 1334 having a size of 2N×nD, apartition mode 1336 having a size of nL×2N, and a partition mode 1338having a size of nR×2N.

Splitting information (TU size flag) of a transformation unit is a typeof a transformation index. The size of the transformation unitcorresponding to the transformation index may be changed according to aprediction unit type or partition mode of the coding unit.

For example, when the partition mode is set to be symmetrical, i.e. thepartition mode 1322, 1324, 1326, or 1328, a transformation unit 1342having a size of 2N×2N is set if a TU size flag of a transformation unitis 0, and a transformation unit 1344 having a size of N×N is set if a TUsize flag is 1.

When the partition mode is set to be asymmetrical, i.e., the partitionmode 1332, 1334, 1336, or 1338, a transformation unit 1352 having a sizeof 2N×2N is set if a TU size flag is 0, and a transformation unit 1354having a size of N/2×N/2 is set if a TU size flag is 1.

Referring to FIG. 23, the TU size flag is a flag having a value of 0 or1, but the TU size flag is not limited to 1 bit, and a transformationunit may be hierarchically split while the TU size flag increases from0. Splitting information (TU size flag) of a transformation unit may bean example of a transformation index.

In this case, the size of a transformation unit that has been actuallyused may be expressed by using a TU size flag of a transformation unit,according to an exemplary embodiment, together with a maximum size andminimum size of the transformation unit. The video encoding apparatus100 is capable of encoding maximum transformation unit size information,minimum transformation unit size information, and a maximum TU sizeflag. The result of encoding the maximum transformation unit sizeinformation, the minimum transformation unit size information, and themaximum TU size flag may be inserted into an SPS. The video decodingapparatus 200 may decode video by using the maximum transformation unitsize information, the minimum transformation unit size information, andthe maximum TU size flag.

For example, (a) if the size of a current coding unit is 64×64 and amaximum transformation unit size is 32×32, (a-1) then the size of atransformation unit may be 32×32 when a TU size flag is 0, (a-2) may be16×16 when the TU size flag is 1, and (a-3) may be 8×8 when the TU sizeflag is 2.

As another example, (b) if the size of the current coding unit is 32×32and a minimum transformation unit size is 32×32, (b-1) then the size ofthe transformation unit may be 32×32 when the TU size flag is 0. Here,the TU size flag cannot be set to a value other than 0, since the sizeof the transformation unit cannot be less than 32×32.

As another example, (c) if the size of the current coding unit is 64×64and a maximum TU size flag is 1, then the TU size flag may be 0 or 1.Here, the TU size flag cannot be set to a value other than 0 or 1.

Thus, if it is defined that the maximum TU size flag is‘MaxTransformSizelndex’, a minimum transformation unit size is‘MinTransformSize’, and a transformation unit size is ‘RootTuSize’ whenthe TU size flag is 0, then a current minimum transformation unit size‘CurrMinTuSize’ that can be determined in a current coding unit, may bedefined by Equation (1):CurrMinTuSize=max(MinTransformSize,RootTuSize/(2^MaxTransformSizeIndex))  (1)

Compared to the current minimum transformation unit size ‘CurrMinTuSize’that can be determined in the current coding unit, a transformation unitsize ‘RootTuSize’ when the TU size flag is 0 may denote a maximumtransformation unit size that can be selected in the system. In Equation(1), ‘RootTuSize/(2^MaxTransformSizeIndex)’ denotes a transformationunit size when the transformation unit size ‘RootTuSize’, when the TUsize flag is 0, is split a number of times corresponding to the maximumTU size flag, and ‘MinTransformSize’ denotes a minimum transformationsize. Thus, a smaller value from among‘RootTuSize/(2^MaxTransformSizeIndex)’ and ‘MinTransformSize’ may be thecurrent minimum transformation unit size ‘CurrMinTuSize’ that can bedetermined in the current coding unit.

According to an exemplary embodiment, the maximum transformation unitsize RootTuSize may vary according to the type of a prediction mode.

For example, if a current prediction mode is an inter mode, then‘RootTuSize’ may be determined by using Equation (2) below. In Equation(2), ‘MaxTransformSize’ denotes a maximum transformation unit size, and‘PUSize’ denotes a current prediction unit size.RootTuSize=min(MaxTransformSize,PUSize)  (2)

That is, if the current prediction mode is the inter mode, thetransformation unit size ‘RootTuSize’, when the TU size flag is 0, maybe a smaller value from among the maximum transformation unit size andthe current prediction unit size.

If a prediction mode of a current partition unit is an intra mode,‘RootTuSize’ may be determined by using Equation (3) below. In Equation(3), ‘PartitionSize’ denotes the size of the current partition unit.RootTuSize=min(MaxTransformSize,PartitionSize)  (3)

That is, if the current prediction mode is the intra mode, thetransformation unit size ‘RootTuSize,’ when the TU size flag is 0, maybe a smaller value from among the maximum transformation unit size andthe size of the current partition unit.

However, the current maximum transformation unit size ‘RootTuSize’ thatvaries according to the type of a prediction mode in a partition unit isjust an example, and the exemplary embodiments are not limited thereto.

According to the video encoding method based on coding units having atree structure as described with reference to FIGS. 11 through 23, imagedata of the space domain is encoded for each coding unit of a treestructure. According to the video decoding method based on coding unitshaving a tree structure, decoding is performed for each LCU toreconstruct image data of the space domain. Thus, a picture and a videothat is a picture sequence may be reconstructed. The reconstructed videomay be reproduced by a reproducing apparatus, stored in a storagemedium, or transmitted through a network.

The exemplary embodiments may be written as computer programs and may beimplemented in general-use digital computers that execute the programsusing a computer-readable recording medium. Examples of thecomputer-readable recording medium include magnetic storage media (e.g.,ROM, floppy discs, hard discs, etc.) and optical recording media (e.g.,CD-ROMs, or DVDs).

For convenience of description, the inter-layer video encoding methodand/or the video encoding method described above with reference to FIGS.1A through 23, will be referred to as a ‘video encoding method accordingto the inventive concept’. In addition, the inter-layer video decodingmethod and/or the video decoding method described above with referenceto FIGS. 1A through 23, will be referred to as a ‘video decoding methodaccording to the inventive concept.’

A video encoding apparatus including the inter-layer video encodingapparatus 10, the video encoding apparatus 100, or the image encoder400, which is described above with reference to FIGS. 1A through 23,will be referred to as a ‘video encoding apparatus according to theinventive concept’. In addition, a video decoding apparatus includingthe inter-layer video decoding apparatus 20, the video decodingapparatus 200, or the image decoder 500, which is described above withreference to FIGS. 1A through 23, will be referred to as a ‘videodecoding apparatus according to the inventive concept’.

A computer-readable recording medium storing a program, e.g., a disc26000, according to an exemplary embodiment will now be described indetail.

FIG. 24 is a diagram of a physical structure of the disc 26000 in whicha program is stored, according to various exemplary embodiments. Thedisc 26000, which is a storage medium, may be a hard drive, a compactdisc-read only memory (CD-ROM) disc, a Blu-ray disc, or a digitalversatile disc (DVD). The disc 26000 includes a plurality of concentrictracks Tr that are each divided into a specific number of sectors Se ina circumferential direction of the disc 26000. In a specific region ofthe disc 26000, a program that executes the quantization parameterdetermination method, the video encoding method, and the video decodingmethod described above may be assigned and stored.

A computer system embodied using a storage medium that stores a programfor executing the video encoding method and the video decoding method asdescribed above will now be described with reference to FIG. 25.

FIG. 25 is a diagram of a disc drive 26800 for recording and reading aprogram by using the disc 26000. A computer system 26700 may store aprogram that executes at least one of a video encoding method and avideo decoding method according to the inventive concept, in the disc26000 via the disc drive 26800. To run the program stored in the disc26000 on the computer system 26700, the program may be read from thedisc 26000 and be transmitted to the computer system 26700 by using thedisc drive 26700.

The program that executes at least one of a video encoding method and avideo decoding method according to an exemplary embodiment may be storednot only in the disc 26000 illustrated in FIG. 24 or 25 but also in amemory card, a ROM cassette, or a solid state drive (SSD).

A system to which the video encoding method and a video decoding methoddescribed above are applied will be described below.

FIG. 26 is a diagram of an overall structure of a content supply system11000 for providing a content distribution service. A service area of acommunication system is divided into predetermined-sized cells, andwireless base stations 11700, 11800, 11900, and 12000 are installed inthese cells, respectively.

The content supply system 11000 includes a plurality of independentdevices. For example, the plurality of independent devices, such as acomputer 12100, a personal digital assistant (PDA) 12200, a video camera12300, and a mobile phone 12500, are connected to the Internet 11100 viaan internet service provider 11200, a communication network 11400, andthe wireless base stations 11700, 11800, 11900, and 12000.

However, the content supply system 11000 is not limited to asillustrated in FIG. 24, and devices may be selectively connectedthereto. The plurality of independent devices may be directly connectedto the communication network 11400, not via the wireless base stations11700, 11800, 11900, and 12000.

The video camera 12300 is an imaging device, e.g., a digital videocamera, which is capable of capturing video images. The mobile phone12500 may employ at least one communication method from among variousprotocols, e.g., Personal Digital Communications (PDC), Code DivisionMultiple Access (CDMA), Wideband-Code Division Multiple Access (W-CDMA),Global System for Mobile Communications (GSM), and Personal HandyphoneSystem (PHS).

The video camera 12300 may be connected to a streaming server 11300 viathe wireless base station 11900 and the communication network 11400. Thestreaming server 11300 allows content received from a user via the videocamera 12300 to be streamed via a real-time broadcast. The contentreceived from the video camera 12300 may be encoded using the videocamera 12300 or the streaming server 11300. Video data captured by thevideo camera 12300 may be transmitted to the streaming server 11300 viathe computer 12100.

Video data captured by a camera 12600 may also be transmitted to thestreaming server 11300 via the computer 12100. The camera 12600 is animaging device capable of capturing both still images and video images,similar to a digital camera. The video data captured by the camera 12600may be encoded using the camera 12600 or the computer 12100. Softwarethat performs encoding and decoding video may be stored in acomputer-readable recording medium, e.g., a CD-ROM disc, a floppy disc,a hard disc drive, an SSD, or a memory card, which may be accessible bythe computer.

If video data is captured by a camera built in the mobile phone 12500,the video data may be received from the mobile phone 12500.

The video data may also be encoded by a large scale integrated circuit(LSI) system installed in the video camera 12300, the mobile phone12500, or the camera 12600.

The content supply system 11000 according to an exemplary embodiment mayencode content data recorded by a user using the video camera 12300, thecamera 12600, the mobile phone 12500, or another imaging device, e.g.,content recorded during a concert, and transmit the encoded content datato the streaming server 11300. The streaming server 11300 may transmitthe encoded content data in a type of a streaming content to otherclients that request the content data.

The clients are devices capable of decoding the encoded content data,e.g., the computer 12100, the PDA 12200, the video camera 12300, or themobile phone 12500. Thus, the content supply system 11000 allows theclients to receive and reproduce the encoded content data. Also, thecontent supply system 11000 allows the clients to receive the encodedcontent data and decode and reproduce the encoded content data in realtime, thereby enabling personal broadcasting.

Encoding and decoding operations of the plurality of independent devicesincluded in the content supply system 11000 may be similar to those of avideo encoding apparatus and a video decoding apparatus according to theinventive concept.

The mobile phone 12500 included in the content supply system 11000according to an exemplary embodiment will now be described in greaterdetail with reference to FIGS. 27 and 28.

FIG. 27 illustrates an external structure of the mobile phone 12500 towhich a video encoding method and a video decoding method according tothe inventive concept are applied, according to various exemplaryembodiments. The mobile phone 12500 may be a smart phone, the functionsof which are not limited and a large number of the functions of whichmay be changed or expanded via an application program.

The mobile phone 12500 includes an internal antenna 12510 via which aradio-frequency (RF) signal may be exchanged with the wireless basestation 12000, and includes a display screen 12520 for displaying imagescaptured by a camera 12530 or images that are received via the antenna12510 and decoded, e.g., a liquid crystal display (LCD) or an organiclight-emitting diode (OLED) screen. The mobile phone 12500 includes anoperation panel 12540 including a control button and a touch panel. Ifthe display screen 12520 is a touch screen, the operation panel 12540further includes a touch sensing panel of the display screen 12520. Themobile phone 12500 includes a speaker 12580 for outputting voice andsound or another type of sound outputter, and a microphone 12550 forinputting voice and sound or another type sound inputter. The mobilephone 12500 further includes the camera 12530, such as a charge-coupleddevice (CCD) camera, to capture video and still images. The mobile phone12500 may further include a storage medium 12570 for storingencoded/decoded data, e.g., video or still images captured by the camera12530, received via email, or obtained according to various ways; and aslot 12560 via which the storage medium 12570 is loaded into the mobilephone 12500. The storage medium 12570 may be a flash memory, e.g., asecure digital (SD) card or an electrically erasable and programmableread only memory (EEPROM) included in a plastic case.

FIG. 28 illustrates an internal structure of the mobile phone 12500. Tosystemically control parts of the mobile phone 12500 including thedisplay screen 12520 and the operation panel 12540, a power supplycircuit 12700, an operation input controller 12640, an image encoder12720, a camera interface 12630, an LCD controller 12620, an imagedecoder 12690, a multiplexer/demultiplexer 12680, a recorder/reader12670, a modulator/demodulator 12660, and a sound processor 12650 areconnected to a central controller 12710 via a synchronization bus 12730.

If a user operates a power button and sets from a ‘power off’ state to a‘power on’ state, the power supply circuit 12700 supplies power to allthe parts of the mobile phone 12500 from a battery pack, thereby settingthe mobile phone 12500 in an operation mode.

The central controller 12710 includes a central processing unit (CPU), aROM, and a RAM.

While the mobile phone 12500 transmits communication data to theoutside, a digital signal is generated by the mobile phone 12500 undercontrol of the central controller 12710. For example, the soundprocessor 12650 may generate a digital sound signal, the image encoder12720 may generate a digital image signal, and text data of a messagemay be generated via the operation panel 12540 and the operation inputcontroller 12640. When a digital signal is transmitted to themodulator/demodulator 12660 under control of the central controller12710, the modulator/demodulator 12660 modulates a frequency band of thedigital signal, and a communication circuit 12610 performsdigital-to-analog conversion (DAC) and frequency conversion on thefrequency band-modulated digital sound signal. A transmission signaloutput from the communication circuit 12610 may be transmitted to avoice communication base station or the wireless base station 12000 viathe antenna 12510.

For example, when the mobile phone 12500 is in a conversation mode, asound signal obtained via the microphone 12550 is transformed into adigital sound signal by the sound processor 12650, under control of thecentral controller 12710. The digital sound signal may be transformedinto a transformation signal via the modulator/demodulator 12660 and thecommunication circuit 12610, and may be transmitted via the antenna12510.

When a text message, e.g., email, is transmitted in a data communicationmode, text data of the text message is input via the operation panel12540 and is transmitted to the central controller 12710 via theoperation input controller 12640. Under control of the centralcontroller 12710, the text data is transformed into a transmissionsignal via the modulator/demodulator 12660 and the communication circuit12610 and is transmitted to the wireless base station 12000 via theantenna 12510.

To transmit image data in the data communication mode, image datacaptured by the camera 12530 is provided to the image encoder 12720 viathe camera interface 12630. The captured image data may be directlydisplayed on the display screen 12520 via the camera interface 12630 andthe LCD controller 12620.

A structure of the image encoder 12720 may correspond to that of theabove-described video encoding apparatus according to the inventiveconcept. The image encoder 12720 may transform the image data receivedfrom the camera 12530 into compressed and encoded image data based onthe above-described video encoding method according to the inventiveconcept, and then output the encoded image data to themultiplexer/demultiplexer 12680. During a recording operation of thecamera 12530, a sound signal obtained by the microphone 12550 of themobile phone 12500 may be transformed into digital sound data via thesound processor 12650, and the digital sound data may be transmitted tothe multiplexer/demultiplexer 12680.

The multiplexer/demultiplexer 12680 multiplexes the encoded image datareceived from the image encoder 12720, together with the sound datareceived from the sound processor 12650. A result of multiplexing thedata may be transformed into a transmission signal via themodulator/demodulator 12660 and the communication circuit 12610, and maythen be transmitted via the antenna 12510.

While the mobile phone 12500 receives communication data from theoutside, frequency recovery and ADC are performed on a signal receivedvia the antenna 12510 to transform the signal into a digital signal. Themodulator/demodulator 12660 modulates a frequency band of the digitalsignal. The frequency-band modulated digital signal is transmitted tothe video decoder 12690, the sound processor 12650, or the LCDcontroller 12620, according to the type of the digital signal.

In the conversation mode, the mobile phone 12500 amplifies a signalreceived via the antenna 12510, and obtains a digital sound signal byperforming frequency conversion and ADC on the amplified signal. Areceived digital sound signal is transformed into an analog sound signalvia the modulator/demodulator 12660 and the sound processor 12650, andthe analog sound signal is output via the speaker 12580, under controlof the central controller 12710.

When in the data communication mode, data of a video file accessed at anInternet website is received, a signal received from the wireless basestation 12000 via the antenna 12510 is output as multiplexed data viathe modulator/demodulator 12660, and the multiplexed data is transmittedto the multiplexer/demultiplexer 12680.

To decode the multiplexed data received via the antenna 12510, themultiplexer/demultiplexer 12680 demultiplexes the multiplexed data intoan encoded video data stream and an encoded audio data stream. Via thesynchronization bus 12730, the encoded video data stream and the encodedaudio data stream are provided to the video decoder 12690 and the soundprocessor 12650, respectively.

A structure of the image decoder 12690 may correspond to that of theabove-described video decoding apparatus according to the inventiveconcept. The image decoder 12690 may decode the encoded video data toobtain reconstructed video data and provide the reconstructed video datato the display screen 12520 via the LCD controller 12620, by using theabove-described video decoding method.

Thus, the data of the video file accessed at the Internet website may bedisplayed on the display screen 12520. At the same time, the soundprocessor 12650 may transform audio data into an analog sound signal,and provide the analog sound signal to the speaker 12580. Thus, audiodata contained in the video file accessed at the Internet website mayalso be reproduced via the speaker 12580.

The mobile phone 12500 or another type of communication terminal may bea transceiving terminal including both a video encoding apparatus and avideo decoding apparatus according to the inventive concept, may be atransceiving terminal including only the video encoding apparatus, ormay be a transceiving terminal including only the video decodingapparatus.

A communication system according to the inventive concept is not limitedto the communication system described above with reference to FIG. 28.For example, FIG. 29 illustrates a digital broadcasting system employinga communication system, according to various exemplary embodiments. Thedigital broadcasting system of FIG. 29 according to an exemplaryembodiment may receive a digital broadcast transmitted via a satelliteor a terrestrial network by using a video encoding apparatus and a videodecoding apparatus according to an exemplary embodiment.

Specifically, a broadcasting station 12890 transmits a video data streamto a communication satellite or a broadcasting satellite 12900 by usingradio waves. The broadcasting satellite 12900 transmits a broadcastsignal, and the broadcast signal is transmitted to a satellite broadcastreceiver via a household antenna 12860. In every house, an encoded videostream may be decoded and reproduced by a TV receiver 12810, a set-topbox 12870, or another device.

When a video decoding apparatus according to the inventive concept isimplemented in a reproducing apparatus 12830, the reproducing apparatus12830 may parse and decode an encoded video stream recorded on a storagemedium 12820, such as a disc or a memory card to reconstruct digitalsignals. Thus, the reconstructed video signal may be reproduced, forexample, on a monitor 12840.

In the set-top box 12870 connected to the antenna 12860 for asatellite/terrestrial broadcast or a cable antenna 12850 for receiving acable television (TV) broadcast, a video decoding apparatus according tothe inventive concept may be installed. Data output from the set-top box12870 may also be reproduced on a TV monitor 12880.

As another example, a video decoding apparatus according to theinventive concept may be installed in the TV receiver 12810 instead ofthe set-top box 12870.

An automobile 12920 that has an appropriate antenna 12910 may receive asignal transmitted from the satellite 12900 or the wireless base station11700. A decoded video may be reproduced on a display screen of anautomobile navigation system 12930 installed in the automobile 12920.

A video signal may be encoded by a video encoding apparatus according tothe inventive concept and may then be stored in a storage medium.Specifically, an image signal may be stored in a DVD disc 12960 by a DVDrecorder or may be stored in a hard disc by a hard disc recorder 12950.As another example, the video signal may be stored in an SD card 12970.If the hard disc recorder 12950 includes a video decoding apparatusaccording to an exemplary embodiment, a video signal recorded on the DVDdisc 12960, the SD card 12970, or another storage medium may bereproduced on the TV monitor 12880.

The automobile navigation system 12930 may not include the camera 12530,the camera interface 12630, and the image encoder 12720. For example,the computer 12100 and the TV receiver 12810 may also not include thecamera 12530, the camera interface 12630, and the image encoder 12720.

FIG. 30 is a diagram illustrating a network structure of a cloudcomputing system using a video encoding apparatus and a video decodingapparatus, according to various exemplary embodiments.

The cloud computing system may include a cloud computing server 14000, auser database (DB) 14100, computing resources 14200, and a userterminal.

The cloud computing system provides an on-demand outsourcing service ofthe plurality of computing resources via a data communication network,e.g., the Internet, in response to a request from the user terminal.Under a cloud computing environment, a service provider provides userswith desired services by combining computing resources at data centerslocated at physically different locations by using virtualizationtechnology. A service user does not have to install computing resources,e.g., an application, a storage, an operating system (OS), and security,into his/her own terminal in order to use them, but may select and usedesired services from among services in a virtual space generatedthrough the virtualization technology, at a desired point in time.

A user terminal of a specified service user is connected to the cloudcomputing server 14000 via a data communication network including theInternet and a mobile telecommunication network. User terminals may beprovided cloud computing services, and particularly video reproductionservices, from the cloud computing server 14000. The user terminals maybe various types of electronic devices capable of being connected to theInternet, e.g., a desktop PC 14300, a smart TV 14400, a smart phone14500, a notebook computer 14600, a portable multimedia player (PMP)14700, a tablet PC 14800, and the like.

The cloud computing server 14100 may combine the plurality of computingresources 14200 distributed in a cloud network and provide userterminals with a result of combining. The plurality of computingresources 14200 may include various data services, and may include datauploaded from user terminals. As described above, the cloud computingserver 14100 may provide user terminals with desired services bycombining video database distributed in different regions according tothe virtualization technology.

User information about users who have subscribed for a cloud computingservice is stored in the user DB 14100. The user information may includelogging information, addresses, names, and personal credit informationof the users. The user information may further include indexes ofvideos. Here, the indexes may include a list of videos that have alreadybeen reproduced, a list of videos that are being reproduced, a pausingpoint of a video that was being reproduced, and the like.

Information about a video stored in the user DB 14100 may be sharedbetween user devices. For example, when a video service is provided tothe notebook computer 14600 in response to a reproduction request fromthe notebook computer 14600, a reproduction history of the video serviceis stored in the user DB 14100. When a request to reproduce this videoservice is received from the smart phone 14500, the cloud computingserver 14000 searches for and reproduces this video service, based onthe user DB 14100. When the smart phone 14500 receives a video datastream from the cloud computing server 14000, a process of reproducingvideo by decoding the video data stream is similar to an operation ofthe mobile phone 12500 described above with reference to FIG. 26.

The cloud computing server 14000 may refer to a reproduction history ofa desired video service, stored in the user DB 14100. For example, thecloud computing server 14000 receives a request to reproduce a videostored in the user DB 14100, from a user terminal. If this video wasbeing reproduced, then a method of streaming this video, performed bythe cloud computing server 14000, may vary according to the request fromthe user terminal, i.e., according to whether the video will bereproduced, starting from a start thereof or a pausing point thereof.For example, if the user terminal requests to reproduce the video,starting from the start thereof, the cloud computing server 14000transmits streaming data of the video starting from a first framethereof to the user terminal. If the user terminal requests to reproducethe video, starting from the pausing point thereof, the cloud computingserver 14000 transmits streaming data of the video starting from a framecorresponding to the pausing point, to the user terminal.

In this case, the user terminal may include a video decoding apparatusas described above with reference to FIGS. 1A through 23. As anotherexample, the user terminal may include a video encoding apparatus asdescribed above with reference to FIGS. 1A through 23. Alternatively,the user terminal may include both the video encoding apparatus and thevideo decoding apparatus as described above with reference to FIGS. 1Athrough 23.

Various exemplary embodiments of a video encoding method, a videodecoding method, a video encoding apparatus, and a video decodingapparatus described above with reference to FIGS. 1A through 23 havebeen described above with reference to FIGS. 24 to 30. However, methodsof storing the video encoding method and the video decoding method in astorage medium or methods of implementing the video encoding apparatusand the video decoding apparatus in a device, according to variousexemplary embodiments, described above with reference to FIGS. 1Athrough 23 are not limited to the exemplary embodiments described abovewith reference to FIGS. 24 to 30.

It should be understood that the exemplary embodiments described thereinshould be considered in a descriptive sense only and not for purposes oflimitation. Descriptions of features or aspects within each exemplaryembodiment should typically be considered as available for other similarfeatures or aspects in other exemplary embodiments.

While an exemplary embodiment of the inventive concept have beendescribed with reference to the figures, it will be understood by thoseof ordinary skill in the art that various changes in form and detailsmay be made therein without departing from the spirit and scope of theinventive concept as defined by the following claims.

The invention claimed is:
 1. An inter-layer video decoding method,comprising: obtaining a decoded first layer image from a first layerbitstream; obtaining, residual prediction information from a secondlayer bitstream, wherein the residual prediction information representswhether to perform residual prediction performed on a second layer blockby referring to a first layer block corresponding to the second layerblock within the first layer image; in response to determining not toperform the residual prediction on the second layer block based on theobtained residual prediction information, obtaining luminancecompensation information from the second layer bitstream, wherein theluminance compensation information represents whether to performluminance compensation on the second layer block; and in response todetermining to perform the luminance compensation on the second layerblock based on the obtained luminance compensation information,performing the luminance compensation on the second layer block toreconstruct the second layer block.
 2. The inter-layer video decodingmethod of claim 1, further comprising: in response to determining not toperform the luminance compensation on the second layer block based onthe obtained luminance compensation information, not performing theluminance compensation on the second layer block.
 3. The inter-layervideo decoding method of claim 1, further comprising: in response todetermining to perform the residual prediction on the second layer blockbased on the obtained residual prediction information, performing theresidual prediction to reconstruct the second layer block.
 4. Theinter-layer video decoding method of claim 1, further comprising; inorder to reconstruct the second layer block, a predetermined partitiontype and a predetermined prediction mode of the second layer block aredetermined, determining whether to apply the residual prediction basedon at least one of the predetermined prediction mode and thepredetermined partition type of the second layer block, wherein thedetermining whether to perform the residual prediction comprises, inresponse to determining to apply the residual prediction, determiningwhether to perform the residual prediction based on the residualprediction information.
 5. The inter-layer video decoding method ofclaim 4, wherein the determining of whether to apply the residualprediction comprises: in response to the predetermined partition typeindicating a 2N×2N type, and the predetermined prediction modeindicating a merge mode or a skip mode, applying the residualprediction.
 6. The inter-layer video decoding method of claim 4, whereinthe determining of whether to apply the residual prediction comprises:in response to the predetermined partition type indicating a 2N×2N type,and the predetermined prediction mode indicating a merge mode or a skipmode, and luminance compensation information (ic_flag) obtained from thesecond layer bitstream indicating 0, applying the residual prediction.7. The inter-layer video decoding method of claim 1, wherein residualprediction information obtained from the second layer bitstreamcomprises at least one from among flag information, weight information,prediction direction information, picture type information, and layerinformation.
 8. The inter-layer video decoding method of claim 1,further comprising: in response to determining to perform the residualprediction, determining a weight value to be applied to residual datawhen performing the residual prediction; and performing the residualprediction by applying the weight value to the residual data.
 9. Theinter-layer video decoding method of claim 8, wherein the weight valueis applied to luma residual data among the residual data.
 10. Aninter-layer video encoding method, comprising: encoding a first layerimage and generating a first layer bitstream including generatedencoding information; obtaining residual prediction information, whereinthe residual prediction information represents whether to performresidual prediction performed on a second layer block by referring to afirst layer block corresponding to the second layer block within thefirst layer image; in response to determining not to perform theresidual prediction on the second layer block based on the obtainedresidual prediction information, obtaining luminance compensationinformation which represents whether to perform luminance compensationon the second layer block; and in response to determining to perform theluminance compensation on the second layer block based on the obtainedluminance compensation information, performing the luminancecompensation on the second layer block to encode the second layer block.11. The inter-layer video encoding method of claim 10, in response todetermining not to perform the luminance compensation on the secondlayer block based on the obtained luminance compensation information,not performing the luminance compensation on the second layer block. 12.The inter-layer video encoding method of claim 10, further comprising,in response to determining to perform the residual prediction on thesecond layer block based on the obtained residual predictioninformation, performing the residual prediction to encode the secondlayer block.
 13. The inter-layer video encoding method of claim 10,further comprising; in order to encode the second layer block, apredetermined partition type and a predetermined prediction mode of thesecond layer block are determined, determining whether to apply theresidual prediction based on at least one of the predeterminedprediction mode and the predetermined partition type of the second layerblock, wherein the determining whether to perform the residualprediction comprises, in response to determining to apply the residualprediction, determining whether to perform the residual prediction basedon the residual prediction information.
 14. An inter-layer videodecoding apparatus, comprising: a first layer decoder configured toobtain a decoded first layer image from a first layer bitstream; aresidual prediction determiner configured to obtain, based on residualprediction information from a second layer bitstream, wherein theresidual prediction information represents whether to perform residualprediction performed on a second layer block by referring to a firstlayer block corresponding to the second layer block within the firstlayer image; a luminance compensation determiner configured to, inresponse to determining not to perform the residual prediction on thesecond layer block based on the obtained residual predictioninformation, obtain luminance compensation information from the secondlayer bitstream, wherein the luminance compensation informationrepresents whether to perform luminance compensation based on theluminance compensation information; and a second layer decoderconfigured to perform the luminance compensation on the second layerblock in response to determining to perform the luminance compensationon the second layer block based on the obtained luminance compensationinformation.
 15. An inter-layer video encoding apparatus, comprising: afirst layer encoder configured to encode a first layer image andgenerate a first layer bitstream including generated encodinginformation; a residual prediction determiner configured to obtainresidual prediction information, wherein the residual predictioninformation represents whether to perform residual prediction on asecond layer block by referring to a first layer block corresponding tothe second layer block within the first layer image; a luminancecompensation determiner configured to, in response to determining not toperform the residual prediction on the second layer block based on theobtained residual prediction information, obtain luminance compensationinformation which represents whether to perform luminance compensationon the second layer block, and obtain luminance compensationinformation; and a second layer encoder configured to, in response todetermining to perform the luminance compensation on the second layerblock based on the obtained luminance compensation information, performthe luminance compensation on the second layer block to encode thesecond layer block.